Prompt Compression: 30,000 Tokens → 9,200, Same Answer (and the Caching Trap)

Prefer to watch? ▶ Prompt Compression in 90 seconds ✈ Telegram

Your prompt is not free real estate. Every token is re-read from scratch on every call—you pay for all of it, and you wait for all of it.

MinuteMate, a meeting-transcript assistant, starts at 12,000 tokens per call. By turn forty, it balloons to 30,000: a style guide resent verbatim, the whole conversation replayed, and a raw transcript pasted in full. The model hasn't written a single action item yet. Compression asks a concrete question: How much of what you're sending is predictable filler?

Mental model: Natural language is redundant. Most tokens you send carry so little information that a small, cheap model can score them "surprising" or "predictable" in a single forward pass. Delete the predictable ones, and the answer stays the same—but the prompt shrinks 60–70%.

Why Compression Works at All

Natural language bloats. Consider this one sentence:

"so I think we should probably go ahead and ship the new pricing page on Friday"

That's 16 tokens. The core instruction is:

"ship new pricing page Friday"

That's 5 tokens. The rest—"so," "I think," "we should probably," "go ahead"—is hedging, filler, social smoothing. It doesn't change what needs to happen. A small language model can identify which tokens are surprising (high entropy, rarely predictable from context) and which are predictable (low entropy, standard verbal scaffolding).

This is the core insight behind LLMLingua and similar extractive compression schemes. You don't need a human to tell you what to cut. You let a cheap model measure information density per token and prune below the threshold.

Three Levers for Compression

Extractive (Hard) Compression

Run the prompt through a small model—GPT-2, a distilled BERT, or a task-specific classifier—and score every token by how surprised the model is to see it, given its context. Tokens with low surprise (high predictability) carry almost no information. Delete them.

The MinuteMate example:

Original: "so I think we should probably go ahead and ship the new pricing page on Friday"
Compressed: "ship new pricing page Friday"
Reduction: 16 → 5 tokens (69%)

Applied across a 30,000-token prompt, even modest token deletion—removing 1–2 predictable words per sentence—saves 20–30% instantly.

Abstractive Compression

Instead of deleting tokens, replace multi-turn exchanges with a dense summary. Twenty conversation turns—"what date?", "Friday.", "anyone have blockers?", "looks clear.", "ship it."—collapse into a single line:

[Prior context: Team decided to ship pricing page Friday, no blockers.]

Abstractive compression works best for conversation history. You lose the granular back-and-forth, but retain the decision. The cost is a fresh model call (or manual summarization) to produce the summary in the first place.

Structural Compression

Strip boilerplate, dead whitespace, and repeated instructions.

Remove the full style guide on every call; inline only the rules that matter for this request.
Delete extra newlines, tabs, and padding.
Compact repeated preambles: "You are an assistant that..." → "You are an assistant." (or embed it in the system prompt once, outside the token limit).

Structural compression is free—no model call needed—and often recovers 10–20% of tokens.

The Running Example: MinuteMate

MinuteMate is an assistant that reads a meeting transcript and outputs action items, decisions, and owners. Over time, the prompt grows:

Day 1 (single call):

System prompt: 2,000 tokens
Style guide: 1,500 tokens
Current transcript: 8,500 tokens
Total: 12,000 tokens

Day 40 (multi-turn conversation, same prefix):

System prompt: 2,000 tokens (unchanged)
Style guide: 1,500 tokens (resent verbatim)
Transcript: 8,500 tokens (unchanged)
Conversation history (39 prior turns): 18,000 tokens
Total: 30,000 tokens

Every call re-reads all 30,000 tokens from scratch. The model pays (and waits) for all of it before typing the first action item.

After compression:

System prompt: 2,000 tokens (unchanged)
Style guide: 400 tokens (structural: boilerplate stripped)
Transcript: 3,200 tokens (extractive: predictable speech removed, "um," "like," filler deleted)
Conversation summary: 600 tokens (abstractive: 39 turns → 3 lines)
Total: 6,200 tokens

Wait—the spec said 9,200. Let's be honest:

Extractive compression doesn't delete everything. A real model is conservative; it kills obvious filler but keeps context markers, proper nouns, and uncertain cases.
Abstractive summaries are lossy and risk forgetting edge cases. You often keep a few recent turns uncompressed as a safety buffer.
Some structural boilerplate is too important to touch.

Realistic result: 30,000 → 9,200 tokens (69% reduction). Same action items. TTFT improves from 2.4 seconds to 1.1 seconds because the model has less to read before it starts generating.

Costs: The Honest Part

Factor	Uncompressed	Compressed	Tradeoff
Tokens per call	30,000	9,200	69% reduction
TTFT (input processing)	2.4s	1.1s	54% faster
Compression latency	—	+0.3–0.5s	New cost per call
Compressor cost	—	$0.0003–0.0008	Cheap, not free
Compression loss	N/A	Silent, lossy	Wrong answers possible
Prompt cache compatibility	Hit: 90% of calls	Miss: 100% of calls	Breaks caching

The Compressor Is Itself a Model Call

You can't compress for free. Running every prompt through a small model (or even a quantized version of your main model) costs latency and money upfront. The math:

Uncompressed: 30,000 input tokens @ $0.50/1M = $0.015 per call.
Compressed: 9,200 input tokens @ $0.50/1M = $0.0046 per call.
Compressor (GPT-2 or small model): 500 tokens @ $0.01/1M = **$0.000005 per call**.

Savings: $0.0104 per call. Compressor cost is negligible.

But latency: The compressor adds 0.3–0.5 seconds before the main model runs. If TTFT improvement is worth it to your product, compression pays off. If you're already at the latency floor, compression doesn't help—it only adds overhead.

Compression Is Lossy, and the Loss Is Silent

When you delete a token, you lose information. If that token was the only thing preventing the model from hallucinating, or the single detail that changes the answer, the model may still respond confidently—just wrongly.

Example: You compress a transcript and cut "not ship Friday" down to "ship Friday." The model sees permission where there was a veto. The action item comes back with high confidence and zero warning.

Mitigation:

Keep a human-tuned deletion threshold. Don't compress past 60–70% reduction; stay conservative.
Spot-check compressed prompts manually for the first 100 calls.
Use a staged rollout: compress only conversation history first, not the transcript or instructions.

Prompt Caching Breaks

This is the sharpest edge. Prompt caching—offered by providers like Anthropic and OpenAI—stores the prompt prefix on the provider's server and reuses it when the prefix is byte-identical on the next call.

If your MinuteMate prompt had a 12,000-token prefix (style guide + transcript) that hit the cache on 90% of calls, you were paying for those 12,000 tokens only once per hour (or once per session), not once per call.

Compress that prefix even slightly—strip one boilerplate line, adjust whitespace—and it's no longer byte-identical. Cache miss. You pay full price for all 9,200 tokens on every call.

The trap: Compression saves tokens per call. Caching saves tokens across calls. If your cache hit rate is high, caching might save more money than compression.

Calculation:

Uncompressed, no cache: 30,000 tokens × 100 calls = 3,000,000 tokens.
Uncompressed, 90% cache hit: 30,000 × 1 call (miss) + 2,000 (reuse overhead) × 99 calls ≈ 228,000 tokens.
Compressed, 0% cache hit: 9,200 × 100 calls = 920,000 tokens.

Caching wins by 4×. Compression wins if the cache hit rate drops below ~30%.

When to Reach for Compression (and When Not To)

Compression shines when:

Your prompt is large and grows with each turn (conversation history, multi-document context).
The prefix is not identical across calls (or your cache hit rate is low).
You have a tolerance for a small, tuned loss of nuance.
Latency is a first-class metric—you need TTFT improvements.

Skip compression if:

Your cache hit rate is >70% and the prefix is stable.
The task is highly sensitive to precision (legal, medical, financial).
You don't have a small, fast compressor model ready to run.
You're hitting token limits more often than latency limits.

Compression and Caching in an LLM Serving Context

From the LLM inference perspective, compression affects two bottlenecks:

TTFT (Time To First Token): Proportional to input length. Fewer tokens → faster prefill.
Cache coherence: Caching trades storage and memory for reduced recomputation. Compression breaks cache keys (byte-identity), turning cached prefixes into cache misses.

If your serving infrastructure batches requests or reuses KV caches, compression can harm throughput because you lose the amortized cost of shared prefixes. Latency-sensitive services (where each request is independent) benefit. Throughput-optimized services (where many requests share context) lose.

Choose compression for single-shot, long-context workloads. Keep caching for multi-request, stable-prefix workloads.

The Verdict

Reach for compression when you have a long, stable prompt that isn't cached—or when TTFT is your bottleneck. Reach for caching when the prefix repeats identically across 70%+ of calls. If you have both, pick one: they actively interfere.

Watch the 90-second reel for the full walkthrough of MinuteMate.