Speculative Decoding: A Small Model Drafts, a Big Model Approves

Speculative decoding lets a large language model produce more than one token per forward pass without altering the output distribution. A small draft model proposes a short continuation, then the big target model scores all proposed tokens in parallel, accepting good guesses and replacing the first bad one. The mechanism relies on a modified rejection sampler that guarantees the same distribution as plain autoregressive decoding. The real benefit is wall-clock speed: one trip through the target's weights yields several tokens in memory-bound serving, and the original papers measured decoding 2 to 3 times faster.

The trick is counterintuitive because adding an extra model seems like extra work. Yet when GPU (graphics processing unit) time is dominated by reading weights rather than crunching numbers, a draft model's cost is small, and verifying several tokens at once is nearly free alongside the first token's verification. The result: your slowest step becomes your most efficient step.

What problem does speculative decoding actually solve?

A large transformer in inference spends most of its time moving weights from GPU memory into compute units, not doing math. Generating one token requires reading every parameter of every layer once, even though the prefix's keys and values are cached. That serial dependence means you pay the full memory-bandwidth bill per token, and you cannot start the next token's reads before the current token is known. The GPU's arithmetic units sit underused.

Speculative decoding attacks this by letting the expensive target model process multiple token positions in one parallel pass. You only need to read the weights once to score, say, four draft tokens plus one more position. If most of those tokens end up accepted, you advance the sequence by several tokens at roughly the cost of a single autoregressive step. vLLM's guide accordingly presents Speculative Decoding as a way to reduce inter-token latency for memory-bound workloads at medium to low request rates.

So the problem is not CPU (central processing unit) cycles. It is the ratio between memory bandwidth and the number of tokens you can get out of one full model invocation. When that ratio is low, speculative decoding buys you tokens without extra bandwidth.

How does the draft-and-verify loop work?

We have a big target model $p$ and a small, cheap draft model $q$. Both share the same tokenizer. An iteration starts with the current prefix $x_{1:i}$ that we have already generated.

1. The draft model runs autoregressively to propose a short speculative continuation $y_{i+1}, y_{i+2}, \dots , y_{i+K}$. Because the draft is small, this is fast.
2. We concatenate the prefix with all draft tokens and feed the full sequence into the target model in a single forward pass. The target produces a probability distribution for every position from $i+1$ through $i+K+1$: one for each draft token and one for the position after the last draft.
3. Starting from the first speculative token $y_{i+1}$, we compare the draft's probability $q(y_{i+1} \mid x_{1:i})$ to the target's probability $p(y_{i+1} \mid x_{1:i})$ using an acceptance rule. If the token is accepted, we keep it and check the next draft token. At the first rejection we stop: we draw one replacement token from a residual distribution built from $p$ and $q$, throw away the rest of the draft, and the round ends.
4. If all $K$ drafts are accepted, the target's prediction at position $i+K+1$, computed in the same pass, gives one bonus token.

Either way every round adds at least one token: the replacement token after a rejection, or the bonus token after a clean sweep. If all $K$ tokens are accepted, that single target pass gives us $K+1$ tokens of progress. The acceptance probabilities are designed so that the final sequence has the same distribution as if we had used the target model naively, token by token (Leviathan et al. 2022, Fast Inference from Transformers via Speculative Decoding; Chen et al. 2023, Accelerating Large Language Model Decoding with Speculative Sampling). The two teams, at Google and DeepMind, published it within a few months of each other.

Here is the loop in a diagram:

Why is this statistically lossless?

The guarantee rests on a modified rejection sampler. Assume we have a single token position where the target distribution is $p$ and the draft distribution is $q$. The draft proposes a token $x$ with probability $q(x)$. We then accept it with probability

If rejected, we sample a new token from a residual distribution proportional to $p(x) - q(x)A(x)$, which works out to $\max(0,\, p(x) - q(x))$: only the tokens the target wanted more than the draft did. Because the overall chance of finally outputting a token $x$ is exactly $p(x)$, counting both accepted proposals and the resampling fallback, the output distribution matches the target model's.

A worked token makes this concrete. Say the draft gave "Paris" probability 0.9 and the target gives it 0.6. The draft proposes "Paris", and we accept it with probability $0.6 / 0.9 \approx 67\%$. Had the target given "Paris" 0.95, more than the draft's 0.9, we would accept it every time. If "Paris" is rejected, the replacement comes only from tokens where the target's probability is higher than the draft's, which puts back exactly the probability the draft over-spent on "Paris". With greedy decoding (temperature 0), each distribution puts all its weight on its top token, so the rule reduces to: accept if the draft's token equals the target's top token, otherwise output the target's top token.

In the sequential setting, this logic is applied left to right over the speculative suffix, stopping at the first rejection. At each step, the target's probabilities are conditioned on the prefix that includes the previously accepted tokens. Both papers prove that this procedure, with exact arithmetic, yields sequences drawn from exactly the target model's distribution.

In practice, floating-point differences can cause tiny deviations. vLLM's documentation says its implementation is validated to be lossless up to the precision limits of hardware numerics, while warning that floating-point precision and batch size can still make outputs differ slightly with and without speculation. For almost every application, that means you can turn it on without changing what the model would have said.

Where does the real speedup come from?

The wall-clock gain is not from doing less total computation. The target model still scores all positions, including the drafts it rejects. The gain comes from how the GPU spends its time.

When a large model runs one token at a time, the GPU's arithmetic units sit idle while parameters stream in from memory. Processing $K$ tokens in one shot loads the weights once and reuses them across multiple token positions via batched matrix multiplications. Chen et al. build on exactly this observation: scoring a short continuation in parallel takes about as long as sampling a single token from the target. The draft model adds its own cost, but that cost is a fraction of the target model's because the draft is much smaller (Chen et al. used a 4B draft for the 70B Chinchilla; Leviathan et al. used T5-small for T5-XXL).

Thus the measured speedup depends on three factors: the draft model's speed relative to the target's, the number of tokens accepted per pass, and whether the overall serving setup is memory-bound. In low-throughput, single-request regimes, the speedup is largest. In heavily batched throughput-oriented serving where the GPU is compute-bound, the benefit shrinks and can vanish, because the target model's compute already keeps the chip busy and rejected drafts become real wasted work.

How does the KV cache tango work?

With a separate draft model, both models maintain their own KV (key-value) caches. The draft model's cache holds keys and values for the prefix plus any draft tokens it generates during its autoregressive phase. When the target model verifies, it also needs the KV cache for the entire proposed sequence, but it cannot reuse the draft model's cache because the models have different layers and weights.

The target model's cache for the prefix is available from previous steps. When the target processes the concatenated sequence, it computes new keys and values for the draft-token positions and writes them into its own cache; the entries for rejected tokens must be thrown away (or overwritten) before the next round, so only accepted tokens remain. This bookkeeping adds some memory and complexity but is well worth the extra tokens per target pass. The draft model's own KV cache can be managed with the same paged memory techniques used for the target.

What happens when the draft model is wrong?

Acceptance rates determine efficiency. If the draft proposes a token that the target finds much less likely than the draft did, the acceptance probability becomes tiny. In the worst case, the very first draft token is rejected, the algorithm immediately samples a replacement from the residual distribution, and the round still produces one token of progress: that replacement token. So even a completely misaligned draft never stalls the pipeline; you just waste the draft model's computation.

In realistic settings, a smaller model from the same family drafts well. Leviathan et al. measured per-token acceptance rates between 0.53 and 0.89 for such pairs, mostly between 0.6 and 0.8: for example 0.75 for T5-small drafting for T5-XXL on translation with greedy decoding, and 0.74 for an 8B LaMDA drafting for the 137B LaMDA. If each draft token is accepted with probability $\alpha$, the expected number of tokens produced per pass is

where $K$ is the draft length. With $K = 4$ and $\alpha = 0.7$, that is $(1 - 0.7^5)/(1 - 0.7) = 0.832/0.3 \approx 2.8$ tokens per pass on average. When $\alpha$ is near 1, the system approaches the ideal $K+1$ tokens per pass.

Draft quality is the main tunable knob. Better drafts come from a small version of the same model family, or from cheap drafters attached to the target itself. vLLM's guide says model-based drafters give the best latency reduction, and two of them are now supported by both vLLM and TensorRT-LLM: EAGLE-3 heads, which draft from the target's own hidden features (EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test), and multi-token prediction (MTP) modules trained with the model, which DeepSeek describes repurposing for speculative decoding (DeepSeek-V3 Technical Report). Frameworks make it easy to experiment with different drafters without retraining the target.

Quick Reference

The chart collects the measured figures from the two original papers linked above, Fast Inference from Transformers via Speculative Decoding and Accelerating Large Language Model Decoding with Speculative Sampling.

Frequently Asked Questions

Q: Does speculative decoding change the output distribution in ways that matter for fairness or calibration?
No. The algorithm is a modified rejection sampler that, under exact arithmetic, draws from exactly the same distribution as autoregressive decoding from the target model. Floating-point differences remain, as with any change to how the computation is batched, but they are tiny.

Q: Could I use the target model itself as the draft model?
Not as a full copy: you would pay the target model's full cost per draft token, eliminating the memory-bandwidth benefit. But self-speculation adds cheap drafters to the target itself. Medusa adds extra heads that guess several tokens ahead (Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads); EAGLE drafts from the target's hidden features; early-exit methods such as LayerSkip draft with the first few layers and verify with the rest (LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding). Only proper rejection sampling, or exact matching under greedy decoding, keeps the output identical; Medusa's optional "typical acceptance" mode trades that exactness for speed.

Q: What is the memory overhead of running two models?
The draft model adds its own parameters and its own KV cache. A draft one tenth the size of the target adds about a tenth to the weight memory, and its KV cache is usually much smaller than the target's too, since it has fewer and narrower layers. Drafting heads such as EAGLE's or MTP modules are smaller still. In latency-bound scenarios this is usually well worth it.

Q: How do I choose the draft length $K$?
Each extra draft token is less likely to be reached, because every earlier draft must be accepted first, while the drafting cost grows with $K$. So returns diminish: Chen et al. found the overall speedup plateaus or even regresses as $K$ grows, with the best latency at $K = 3$ for one of their tasks. Production frameworks tune $K$ empirically; small values are typical. If your draft model is very accurate, you can push higher.

Q: When is speculative decoding not worth it?
When the serving setup is already compute-bound (high batch sizes, tensor parallelism saturating the GPU) or when the draft model is so slow relative to the target that the extra step erases the gains. Also, if the target model is small enough that memory bandwidth isn't the bottleneck, speculative decoding may not help.

Test yourself

A production serving system uses speculative decoding with a draft model that achieves a per-token acceptance probability of 0.7. The draft length is set to 4, and when all four drafts are accepted the target adds one bonus token. On average, how many tokens does each iteration produce, and what does that imply for speed relative to vanilla decoding if one draft step costs 5% of a target pass?

Answer:
The round produces 1 token if the first draft is rejected (chance 0.3), 2 if the second is the first rejection (0.7 × 0.3 = 0.21), then 3 (0.147), 4 (0.1029), and 5 when all four are accepted plus the bonus (0.7⁴ = 0.2401). The expected value is $\sum_{j=0}^{4} \alpha^j = (1 - 0.7^5)/(1 - 0.7) = 0.832/0.3 \approx 2.77$ tokens per target pass, of which 1.77 are accepted drafts. Vanilla decoding gets exactly 1. But drafting is not free: four draft steps at 5% each make a round last 1.2 target passes, so the speedup is $2.77 / 1.2 \approx 2.3\times$, which is Leviathan et al.'s walltime formula. If vanilla decoding gave 30 tokens per second, speculative decoding would give about 69, as long as the workload stays memory-bound.

Next: The Economics of Inference: Why API Prices Are What They Are. It turns everything so far, prefill, decode, batching and memory, into the price you pay per million tokens.

If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.

Sources

• Fast Inference from Transformers via Speculative Decoding (Leviathan, Kalman and Matias, Google, 2022)
• Accelerating Large Language Model Decoding with Speculative Sampling (Chen et al., DeepMind, 2023)
• Speculative Decoding (vLLM documentation)
• Speculative Decoding (TensorRT-LLM documentation)
• Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (Cai et al. 2024)
• EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (Li et al. 2024)
• EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test (Li et al. 2025)
• LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding (Elhoushi et al. 2024)
• DeepSeek-V3 Technical Report (DeepSeek-AI 2024)