Context Windows Are Not What You Think
A million tokens on the label doesn't mean a million tokens of attention. Lost-in-the-middle, explained.
A context window is not a buffer of text. It is the KV (key-value) cache: a set of per-layer key and value tensors stored in GPU (graphics processing unit) high-bandwidth memory that the model attends over during autoregressive generation. Its size is constrained by HBM capacity and memory bandwidth. Its structure is shaped by positional encoding and attention head design. A larger number on the spec sheet does not mean uniform recall across that span.
Lost in the middle is not a bug. It is the predictable consequence of distance decay in rotary position embeddings, compounded by the fact that attention weights get spread thinner as sequence length grows. If you build long-context applications without understanding the physical substrate, you will ship systems that silently fail in the middle of long documents.
What Physically Is a Context Window?
The context window is the portion of GPU HBM devoted to storing key and value projections for every token the model has seen. These projections are computed once during prefill, cached, and reused across every subsequent decode step.
When you send a prompt, the tokenizer converts it to integer IDs. Each ID becomes an embedding vector. That vector passes through every transformer layer, where learned projection matrices map it to queries, keys, and values for each attention head source. The keys and values get stored. The queries get discarded after computing attention for the current step.
Everything the model "remembers" about your prompt lives in those key and value tensors. Nothing else persists across timesteps. The MLP weights are static. The embedding table is static. Only the KV cache represents dynamic state.
This means context is finite, biased, and dilutable RAM (random-access memory). It is not a tape the model reads sequentially. It is an addressable memory where attention routing determines what information reaches the output. If a fact sits in the KV cache but attention never routes strongly to it, that fact is functionally invisible source.
How Does the KV Cache Actually Work?
During autoregressive generation, token T only needs to attend to tokens 1 through T-1. Without caching, you would recompute keys and values for the entire prefix at every step, turning each decode into an O(n²) operation over the growing sequence.
The KV cache eliminates that redundancy. After computing keys and values for a token, you append them to per-layer tensors and never recompute them. When generating token T, the model computes only the new query, key, and value. It multiplies the new query against all cached keys. The new key and value get appended to the cache source.
The projection cost per step becomes O(1) with respect to sequence length. The attention scan remains O(n) because you still stream over all cached keys and values. But that scan is memory bandwidth bound, not compute bound. The GPU spends most of its time waiting for data to arrive from HBM source.
This split creates two distinct performance regimes. Prefill is compute bound. It processes the entire prompt at once, paying O(n²) attention and filling the KV cache. Decode is memory bandwidth bound. It generates one token at a time, streaming over increasingly large key and value tensors. Time to first token depends on prompt length and FLOPs (floating-point operations). Tokens per second depends on HBM bandwidth and cache size.
Why Does Context Length Hit a Hardware Wall?
KV cache memory scales linearly with sequence length and batch size. For a model with L layers, B sequences in a batch, S tokens per sequence, N_kv KV heads, and head dimension d_h stored in FP16, the cache footprint is approximately:
The factor of 2 at the front accounts for keys and values. The final 2 is bytes per FP16 element source.
Model weights dominate HBM at short contexts. At long contexts with concurrent requests, the KV cache dominates. A 70B parameter model takes roughly 140GB in FP16. A single 128K-token sequence with 8 KV heads and 128-dimensional heads across 80 layers adds roughly 40GB of KV cache. Run 4 such sequences concurrently and you have exhausted an A100's 80GB of HBM before accounting for weights.
This is why serving throughput is often KV-memory-limited, not FLOP-limited. The maximum batch size you can sustain at target latency is set by how many KV caches fit in HBM source. Expanding the nominal context window without changing hardware or compression strategies means reducing batch size. Throughput drops.
How Do Serving Systems Stretch the Limits?
Three techniques dominate production KV cache management: paged attention, KV quantization, and grouped-query attention.
Paged attention treats the KV cache like virtual memory. Instead of allocating one contiguous block per sequence at max context length, it stores keys and values in fixed-size pages and maintains a block table mapping logical positions to physical pages source. This eliminates fragmentation, enables dynamic growth, and allows prefix sharing between sequences that share a common prompt. If two requests use the same system prompt, the pages for that prefix are mapped once and referenced by both block tables.
KV quantization compresses keys and values to 8-bit or 4-bit representations. The attention mechanism is relatively robust to small perturbations in key and value vectors. Moderate quantization often preserves model behavior well, cutting cache usage by half or more at negligible accuracy cost source.
Grouped-query attention reduces the number of KV heads relative to query heads. In standard multi-head attention, every query head has its own key and value head. In GQA, multiple query heads share one pair of KV heads. This reduces the N_kv term in the memory equation directly, shrinking the per-token cache footprint source. Most modern LLMs targeting inference-heavy deployment use GQA. It costs little in quality and buys substantial memory headroom.
What Is Lost in the Middle?
Lost in the middle is the empirical finding that models retrieve information from the beginning and end of a long context much more reliably than from the middle. The recall curve is U-shaped: high accuracy at low position indices, a trough in the middle, then recovery at high indices source.
This is not a training artifact that someone forgot to fix. It is a direct consequence of how rotary position embeddings interact with attention over long distances.
RoPE encodes position by rotating query and key vectors in embedding space. Position P gets a rotation matrix R_p. The attention score between position P and position Q depends on their relative distance because the dot product of rotated vectors decays as the angular gap grows source.
The rotation frequencies are calibrated for the training context length, typically 4K to 8K tokens. At those distances, the decay is manageable. Stretch to 128K tokens and the middle of the sequence is far enough from both ends that attention logits to mid-context tokens become very weak. The model can still see those positions. The keys and values are in the cache. But the attention weights routed to them are so diluted by the long tail of other positions that they contribute almost nothing to the output.
This interacts with causal masking. Early tokens can attend only to earlier tokens. Late tokens can attend to everything. The end of the context therefore receives attention from the query at every generation step. The beginning receives attention from everything that follows it. The middle gets neither privilege. It is attended to by roughly half the sequence, and each attention weight is small.
Why Does Positional Encoding Shape Attention Topology?
RoPE creates distance decay. ALiBi, another relative encoding scheme, adds a linear bias to attention scores that penalizes distance directly. Each attention head gets a fixed slope, and the bias term subtracts that slope multiplied by the relative distance between positions source.
The practical difference is that ALiBi penalizes distance uniformly. RoPE penalizes it in a frequency-dependent way, with high-frequency components losing alignment faster than low-frequency ones. Both produce decay, but the shape differs.
Neither scheme makes the context window a flat search space. The model does not scan over all tokens with equal fidelity. Positional encoding imposes a topology on attention: closer tokens get stronger weights by default. Content can override this through high dot-product similarity, but the positional bias always pulls toward locality.
When you extend context beyond what the encoding was designed for, the topology breaks. RoPE rotations wrap around or saturate. Interpolated positions lose precision. Extrapolation without adjustment causes attention to collapse toward the ends because mid-sequence positions are effectively at distances the model has no mechanism to represent cleanly source.
How Do Long-Context Training and Fine-Tuning Help?
Training on longer sequences does not eliminate lost in the middle. It shifts the position and depth of the trough but does not flatten the curve source.
What long-context fine-tuning does is adjust the model's positional priors. By seeing tokens at larger relative distances during training, the attention mechanism learns to preserve higher weights to mid-range positions. The decay still exists. It just decays more slowly.
Methods like YaRN extend RoPE by interpolating position indices and adjusting frequencies per dimension, effectively stretching the trained rotary schedule to cover longer ranges without breaking source. This is a mathematical fix to the encoding, not a change to the attention mechanism itself. It makes long-range attention scores numerically possible where they would otherwise underflow or saturate.
These techniques make context extension feasible. They do not make the context window uniform. The U-shaped curve persists even in models fine-tuned on 128K or 256K tokens. The middle is always worse.
What About Offloading and Retrieval-Based KV Reduction?
When HBM is not enough, KV caches can be offloaded to CPU (central processing unit) RAM. The problem is bandwidth. PCIe transfers run at a fraction of HBM speed. Naive offload turns every attention step into a transfer bottleneck.
A²ATS, a heterogeneous inference architecture, addresses this by learning compressed key representations that stay on GPU while the full KV cache lives on CPU source. It uses windowed rotary position embeddings to decouple position from content in the compressed keys. Vector quantization approximates attention scores from the compressed representations. During decode, the system retrieves only the top-K most relevant KV entries from CPU, avoiding a full cache scan over the interconnect.
This is KV cache as tiered memory. The GPU holds a lossy index. The CPU holds the full data. Attention becomes a retrieval operation: identify what matters from the index, then fetch only those entries. It is a pragmatic acknowledgment that the context window is a physical resource, not a logical abstraction.
Quick Reference
| Property | Value |
|---|---|
| KV cache shape | [layers, batch, seq_len, kv_heads, head_dim] |
| Memory scaling | Linear in seq_len × batch |
| Dominant bottleneck at long context | HBM capacity and bandwidth |
| Prefill complexity | O(n²) compute bound |
| Decode complexity | O(n) memory bandwidth bound |
| RoPE distance effect | U-shaped attention decay |
| Lost in the middle peak recall position | Start and end of context |
| GQA memory reduction vs MHA | KV heads / query heads |
| FP16 KV cache per token per layer per head | 2 × 2 × head_dim bytes |
Frequently Asked Questions
Q: If I have a 128K context window, can I just dump 128K tokens of documents in and expect the model to use all of them?
No. The model can attend to all 128K tokens, but attention weights get diluted across the sequence. Information in the middle will be retrieved less reliably than information at the start or end. This is a property of the positional encoding and attention mechanism, not a quality issue with specific models.
Q: Does lost in the middle affect structured prompts differently than free text?
Yes. If you place instructions at both the beginning and end of a long prompt with documents in between, the instructions get the primacy and recency boost and are more likely to be followed correctly. Documents sandwiched in the middle are at highest risk of being ignored or underweighted.
Q: Can I fix lost in the middle by repeating important information?
Partially. Repetition increases the probability that at least one copy lands in a high-attention region. It does not change the underlying topology. A better strategy is to place critical information near the beginning or end of the context and to structure prompts so that the model's query at decode time aligns with the positions you want it to attend to.
Q: Why does my throughput collapse when I increase context length?
Because KV cache memory per request grows linearly with sequence length. At fixed HBM, larger per-request caches mean fewer concurrent requests can fit. If your serving system does not use paged attention or prefix sharing, the memory overhead is even worse due to fragmentation and worst-case preallocation.
Q: Is ALiBi better than RoPE for long contexts?
ALiBi extrapolates more gracefully to sequence lengths unseen during training because the linear bias does not depend on calibrated frequency bands. However, RoPE with interpolation or YaRN-style extension can match or exceed ALiBi on long-context benchmarks. The choice is a design tradeoff in the model architecture, not something you can change at inference time.
If you want this kind of breakdown every week, how real systems actually work under the hood, from GPU memory layouts to attention topology, subscribe to Internals Decoded at internalsdecoded.com.
Sources
- Attention Is All You Need (Vaswani et al.)
- Fast Transformer Decoding (Pope et al.)
- Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al.)
- Rotary Position Embeddings (Su et al.)
- Lost in the Middle (Liu et al.)
- GQA: Training Generalized Multi-Query Transformer Models (Ainslie et al.)
- YaRN (Peng et al.)
- ALiBi (Press et al.)
- A²ATS (Wang et al.)
- FlashAttention (Dao et al.)
- KVQuant (Zandieh et al.)
- arxiv.org · 2306.15595