Speculative Decoding: A Small Model Drafts, a Big Model Approves

Speculative decoding lets a large language model produce more than one token per forward pass without altering the output distribution. A small draft model proposes a short continuation, then the big target model scores all proposed tokens in parallel, accepting correct guesses and resampling any mistakes. The mechanism relies on a modified rejection sampler that guarantees statistical identity to purely autoregressive decoding. The real benefit is wall-clock speed: a single parameter-loading trip yields multiple tokens in memory-bound serving regimes, cutting inter-token latency by half or more.

The trick is counterintuitive because adding an extra model seems like extra work. Yet when GPU (graphics processing unit) time is dominated by reading weights rather than crunching numbers, a draft model’s cost is tiny, and verifying many tokens at once is essentially free alongside the first token’s verification. The result: your slowest step becomes your most efficient step.

What problem does speculative decoding actually solve?

A large transformer in inference spends most of its time moving weights from GPU memory into compute units, not doing math. Generating one token requires reading every parameter of every layer once, even though the prefix hidden states are cached. That serial dependence means you pay the full memory-bandwidth bill per token, and you cannot overlap the next token’s memory reads with the current one’s computation. The GPU sits underutilized.

Speculative decoding attacks this by letting the expensive target model process multiple token positions in one parallel pass. You only need to read the weights once to score, say, four draft tokens plus a bonus token. If most of those tokens end up accepted, you advance the sequence by several tokens at roughly the same cost as a single autoregressive step. The bottleneck shifts from per-token serial weight loading to efficient batch verification. This is why frameworks like vLLM and TensorRT-LLM describe speculative decoding as an optimization for “small batch, underutilized GPU” scenarios vLLM docs, TensorRT-LLM.

So the core problem is not CPU (central processing unit) cycles. It is the ratio between memory bandwidth and the number of tokens you can get out of one full model invocation. When that ratio is low, speculative decoding buys you tokens without extra bandwidth.

How does the draft-and-verify loop work?

We have a big target model (p) and a small, cheap draft model (q). Both share the same tokenizer. An iteration starts with the current prefix (x_{1:i}) that we have already generated.

1. The draft model runs autoregressively to propose a short speculative continuation (y_{i+1}, y_{i+2}, \dots , y_{i+K}). Because the draft is small, this is fast.
2. We concatenate the prefix with all draft tokens and feed the full sequence into the target model in a single forward pass. The target produces logits for every position from (i+1) through (i+K).
3. Starting from the first speculative token (y_{i+1}), we compare the draft’s probability (q(y_{i+1} \mid x_{1:i})) to the target’s probability (p(y_{i+1} \mid x_{1:i})) using an acceptance criterion. If accepted, we append it to the sequence and move to the next draft token with the now-extended prefix. If rejected, we draw a new token from a residual distribution built from (p) and discard the rest of the speculative suffix.
4. After processing all (K) positions, the target model can also sample one extra token from the conditional distribution at (i+K+1), giving one token “for free” even if no draft token is accepted.

The whole verification step costs one target forward pass. If all (K) tokens are accepted, that single pass gives us (K+1) steps of progress. The exact acceptance probabilities are designed so that the final sampled sequence has the same distribution as if we had used the target model naively, token by token Google Speculative Decoding paper, DeepMind Speculative Sampling.

Here is the loop in a diagram:

Why is this statistically lossless?

The guarantee rests on a modified rejection sampler. Assume we have a single token position where the target distribution is (p) and the draft distribution is (q). The draft proposes a token (x) with probability (q(x)). We then accept it with probability

[ A(x) = \min!\left(1,; \frac{p(x)}{q(x)}\right) . ]

If rejected, we sample a new token from a residual distribution proportional to (p(x) - q(x)A(x)). Because the overall chance of finally outputting a token (x) is exactly (p(x)), both from accepted proposals and from the resampling fallback, the output distribution matches the target model’s.

In the sequential setting, this logic is applied left to right over the speculative suffix. At each step, the target model’s output is conditioned on the prefix that includes previously accepted tokens. The DeepMind and Google papers prove that this procedure, under infinite-precision arithmetic, yields joint token sequences drawn from the exact distribution of the target model DeepMind Speculative Sampling, Google Speculative Decoding.

In practice, floating-point differences can cause tiny deviations, but the algorithm is “lossless” in the sense intended by production serving: it does not alter the model’s semantics. That means you can turn speculative decoding on for an application that critically depends on correct probabilities and trust that outputs are still drawn from the original distribution.

Where does the real speedup come from?

The wall-clock gain is not from doing less total computation. The target model still scores all positions. The gain comes from how the GPU spends its time.

When a large model runs one token at a time, the GPU core sits idle while parameters stream in from memory. Processing (K) tokens in one shot loads the weights once and reuses them across multiple token positions via batched matrix multiplications. If the model is memory-bound, computing for (K) positions plus an extra one is almost as fast as computing for only one position vLLM docs. The draft model adds its own cost, but that cost is a fraction of the target model’s because the draft is orders of magnitude smaller.

Thus the measured speedup depends on three factors: the draft model’s speed relative to the target’s cost, the number of tokens accepted per pass, and whether the overall serving setup is memory-bound. In low-throughput, single-request regimes, the speedup is largest. In heavily batched throughput-oriented serving where the GPU is compute-bound, the benefit diminishes because the target model’s compute already keeps the chip busy.

How does the KV cache tango work?

Both models maintain separate KV (key-value) caches. The draft model’s cache holds keys and values for the prefix plus any draft tokens it generates during its autoregressive phase. When the target model verifies, it also needs the KV cache for the entire proposed sequence, but it cannot simply reuse the draft model’s cache because the models have different hidden dimensions and weights.

Serving frameworks typically orchestrate this by duplicating the KV cache management. The target model’s cache for the prefix is available from previous steps. When the target model processes the concatenated sequence, it computes new keys and values for the draft-token positions, writes them into its own cache, and later may discard entries for rejected tokens. vLLM, for example, uses a “proposer-scorer-verifier” pipeline where the draft model populates its cache, the target model scores and verifies, and then the framework reconciles which tokens are kept and updates the target’s cache accordingly vLLM speculative decoding PR (pull request). This dance adds moderate memory and bookkeeping overhead but is well worth the increase in tokens per target forward pass.

The draft model’s own KV cache can be managed with the same paged memory techniques used for the target, avoiding fragmentation and enabling sharing across requests.

What happens when the draft model is wrong?

Acceptance rates determine efficiency. If the draft model proposes a token that has zero or very low probability under (p) relative to (q), the acceptance probability becomes tiny. In the worst case, the very first draft token is rejected, the algorithm immediately falls back to resampling from the target’s distribution, and the target model still generates one token of progress (the resampled token and possibly the bonus token). So even a completely misaligned draft never stalls the pipeline; you just waste the draft model’s computation.

In realistic settings, a draft model that is the same architecture but smaller, or an auxiliary head attached to the target, often achieves 60-80% acceptance rates. That yields an expected number of accepted tokens per pass of roughly ( \frac{1-\alpha^{K+1}}{1-\alpha} ) where (\alpha) is the per-token acceptance probability and (K) is the draft length. With (K = 4) and (\alpha = 0.7), you get about 2.4 tokens per pass on average, more than doubling throughput. When (\alpha) is near 1, the system approaches the ideal (K+1) tokens per pass.

Draft quality is the main tunable knob. Better drafts come from using a small version of the same model family, from self-speculation with early exit, or from multi-token prediction heads like Medusa. Frameworks make it easy to experiment with different draft models without retraining the target.

Quick Reference

Frequently Asked Questions

Q: Does speculative decoding change the output distribution in ways that matter for fairness or calibration?
No. The algorithm is a modified rejection sampler that, under exact arithmetic, draws from exactly the same distribution as autoregressive decoding from the target model. Floating-point differences are negligible.

Q: Could I use the target model itself as the draft model?
That would be pointless for speed: you would pay the target model’s full cost per draft token, eliminating the memory-bandwidth benefit. However, self-speculation techniques like early-exit draft heads (Medusa) use cheap auxiliary heads on the same model to approximate the target’s distribution, effectively turning the target into its own cheap drafter.

Q: What is the memory overhead of running two models?
The draft model adds a modest amount of GPU memory (its parameters and its KV cache). With paged memory, the KV cache can be managed alongside the target’s cache, and the draft model’s size is typically hundreds of millions of parameters, compared to tens of billions for the target. The overhead is usually under 10% of total memory, well worth the speedup in latency-bound scenarios.

Q: How do I choose the draft length (K)?
Larger (K) gives more potential tokens per pass but lowers per-token acceptance probability in practice because later draft tokens are less reliable. Production frameworks tune (K) empirically. Typical values are 4-6. If your draft model is very accurate, you can push higher.

Q: When is speculative decoding not worth it?
When the serving setup is already compute-bound (high batch sizes, tensor-parallelism saturating the GPU) or when the draft model is so slow relative to the target that the extra step erases the gains. Also, if the target model is small enough that memory bandwidth isn’t the bottleneck, speculative decoding may not help.

Test yourself

A production serving system uses speculative decoding with a draft model that achieves a per-token acceptance probability of 0.7. The draft length is set to 4. Assume the target model can also sample one extra token at the end. During a single iteration, how many tokens do we expect to accept, on average, and what does that imply for throughput relative to vanilla decoding?

Answer:
The expected number of tokens per iteration, including the extra token, is given by (\sum_{j=0}^{4} \alpha^j) where (\alpha = 0.7). That sum equals ((1 - 0.7^5)/(1 - 0.7) \approx (1 - 0.168)/0.3 = 2.773). So on average, each target forward pass yields about 2.77 tokens. Without speculative decoding, one pass yields exactly 1 token. Throughput increases by roughly the same factor, assuming the draft model’s cost is negligible and the workload remains memory-bound. If vanilla decoding gave 30 tokens per second, speculative decoding would give about 83 tokens per second. The actual wall-clock speedup may be slightly lower due to draft-model overhead, but on the order of 2.5× is realistic.

In Part 6, we will embed speculative decoding into a real serving stack, showing how vLLM schedules draft-and-verify steps alongside dozens of other requests using continuous batching and priority queues to keep the target model fed and the draft model busy.

If this kind of deep, mechanism-first breakdown keeps you reading, subscribe to Internals Decoded at internalsdecoded.com. Each episode picks apart one system layer so you can reason about it, not just use it.

Sources

• Fast Inference from Transformers via Speculative Decoding, Leviathan et al., Google
• Accelerating Large Language Model Decoding with Speculative Sampling, Chen et al., DeepMind
• vLLM Speculative Decoding Documentation
• TensorRT-LLM GitHub repository
• vLLM speculative decoding proposal and implementation (PR #1919)