Speculative Decoding, Explained: Free LLM Speed With Zero Quality Loss

Autoregressive LLM decoding produces one token per forward pass. A 100-token answer means 100 sequential passes through the entire weight matrix. The arithmetic is trivial; the GPU memory trip is where all the time goes. You are bandwidth bound, not compute bound — the hardware sits idle waiting for data.

One-line mental model: Rent a small model to guess K tokens cheaply; verify all K+1 against the big model in one pass (same weight load); accept matches, correct the first mismatch, discard the rest.

---

Why Decoding Is Bottlenecked on Memory, Not Compute

When you generate the next token, the transformer reads every weight in the model to produce a single logit vector. On a GPU:

• Compute work: a few billion floating-point operations.
• Data movement: 70 billion parameters × (2 to 8 bytes per parameter, depending on precision) loaded from GPU memory, then written back.

On modern hardware, moving that much data takes orders of magnitude longer than computing with it. Each forward pass is a round trip to memory for almost no local work. The GPU's arithmetic units are starved. Increasing batch size helps when you have many prompts to process in parallel, but for a single sequence of generations — the most common case in real-time chat — you cannot hide that latency.

This is the setup that speculative decoding exploits.

---

The Core Idea: Draft Model Proposes, Big Model Verifies

Instead of the big model generating one token at a time, use a smaller, cheaper draft model to propose K candidate tokens in a row:

Input: "def quicksort(arr):"
Draft model (1B params) proposes:
  Token 1: "if"
  Token 2: "len"
  Token 3: "("
  Token 4: "arr"

Now, instead of running the big model (70B) four times — once per proposed token — you run it once and ask it to score all five positions: the original input plus the four draft proposals. The big model's forward pass loads the entire weight matrix whether it generates one token or evaluates four. Verification is nearly free relative to generation.

After that single pass, you compare the big model's chosen token at each position against what the draft model proposed:

• Positions match: accept the draft token and keep going.
• First position diverges: replace it with the big model's token, discard all subsequent draft proposals, and stop (or re-run the draft model from that point).

In the best case, you accept all K proposed tokens and advance K steps with one big-model pass. In the worst case, you accept zero and advance one. Either way, you spent one weight-load trip on the big model.

---

Running Example: CodeCue, an In-Editor Code Assistant

Imagine CodeCue, an IDE plugin that auto-completes code as you type. You're writing Python:

def fibonacci(n):
    if n <= 1:
        return n
    |  # cursor here

Without speculative decoding:

1. Big model (Llama 70B) generates token 1: return
2. Reload weights, generate token 2: fib
3. Reload weights, generate token 3: (
4. Reload weights, generate token 4: n
5. Reload weights, generate token 5: -

Four memory round trips for something as simple and predictable as a function call.

With speculative decoding:

1. Draft model (CodeQwen 1B) quickly proposes: return, fib, (, n, -
2. Big model loads weights once, scores all six positions (original + five proposals).
3. Big model agrees on the first four tokens. On position five, it chooses 1 instead of -.
4. Accept: return fib(n - 1, correct the mismatch, discard anything after.

One big-model weight load instead of five. The output is identical to what Llama 70B would have chosen on its own — the accept rule enforces it mathematically.

---

Mathematical Identity: Why Output Quality Is Preserved

The accept rule is designed so that the joint distribution of accepted tokens matches what the big model would have produced alone. Here's why:

At each position, when the big model disagrees with the draft, you take the big model's token. When they agree, you take the accepted token — but you only got there because the big model would have generated it at that position anyway. The prefix that made it through acceptance is a valid execution path of the big model's sampling procedure.

Unlike quantization (which discards precision to trade quality for speed) or distillation (which accepts training-time accuracy loss), speculative decoding does not trade quality. The output distribution is preserved exactly. You can verify this by running the same sequence through both approaches: they will produce identical token probabilities.

---

What Costs You: Acceptance Rate, VRAM, and Workload Shape

Cost	Impact	Mitigation
Bad draft-model accuracy	Mismatches force rejections; you spend compute on draft tokens you discard. End-to-end latency can be worse than single-model decoding.	Match draft model capacity to the workload. Use a model trained on your domain (e.g., CodeQwen for code).
Two models in VRAM	You need memory for both the draft (1–7B) and big model (70B+). On a single GPU, this is tight or infeasible.	Use smaller draft models (300M–1B). Offload draft to CPU in some setups (not recommended for low-latency).
Workload-dependent gains	Boilerplate code, JSON, schema, structured output: high acceptance. Creative writing, reasoning, brainstorming: low acceptance.	Profile your use case. Disable speculative decoding for open-ended tasks; enable for code, API responses, structured data.
Acceptance rate collapse	If draft and big model disagree frequently (e.g., different tokenizers, training data), you get no speedup or slowdown.	Use a draft model derived from or aligned with the big model.

---

Real-World Performance

Typical wall-clock speedups in production:

• Structured output (JSON, code, boilerplate): 2–4×
• Mixed or conversational: 1.5–2×
• Creative or long-reasoning chains: close to 1× (most drafts rejected)

These gains assume a well-tuned draft model and acceptance rates above 60–70%.

---

LLM Inference Context: TTFT vs. TPOT

In LLM serving, speculative decoding affects time-per-output-token (TPOT) — the latency to generate each successive token after the first. Time-to-first-token (TTFT) remains unchanged because the draft model alone cannot produce coherent output; you still need the big model's first forward pass to seed the sequence.

If you use KV caching and continuous batching in your serving infrastructure, speculative decoding stacks orthogonally: caching reduces redundant computation within a sequence, batching amortizes weight loads across multiple prompts, and speculative decoding removes idle GPU cycles within a single sequence. None of these three optimizations make the others redundant.

---

When to Reach for Speculative Decoding

Reach for speculative decoding when your workload is structured, predictable, and code-heavy (code completion, API response generation, schema filling) and you have enough VRAM for two models. Use single-model decoding when your task is creative, open-ended, or reasoning-heavy (storytelling, math steps, brainstorming), VRAM is tight, or acceptance rates are empirically below 40%.

---

Watch the 90-second reel on software-engineer-blog.com to see the full visual breakdown.