Context Windows Are Not What You Think
A million tokens on the label doesn't mean a million tokens of attention. Lost-in-the-middle, explained.
New to context windows? Start with Context Windows: The Model's Working Memory, part 6 of AI From Zero, which covers what fits in a window and why long chats get weird. This piece goes deeper: what sets the limit, what a long context costs on the hardware, how a model knows where each token sits, and why the number on the label is not the length a model can actually use.
The short version: the context window is the maximum number of tokens a model can attend over in one pass, a limit set mainly by how the model was trained. When the model is served, each token's keys and values are stored in a KV cache in GPU memory, so the practical cost of a long window is set by that cache. And a bigger window does not mean even recall across it: models tend to use the start and end of a long input more reliably than the middle, by amounts that vary a lot from model to model.
What sets the size of a context window?
The context window is the longest sequence a model is built and trained to handle. Two things set it. The first is training: a model learns to use positions and distances only as far as its training sequences went. The second is the cost of long sequences, in compute while training and in memory while serving.
Meta's Llama 3 is a clear example. Its largest model was first pretrained on sequences of 4,096 and then 8,192 tokens (the smaller ones followed a similar recipe), and only "in the final stages of pre-training" on long sequences, raising the length step by step to support 128K tokens. The paper says why: "We do not train on long sequences earlier because the compute in self-attention layers grows quadratically in the sequence length" (The Llama 3 Herd of Models, Meta).
The KV cache is a separate thing. It is an optimization for generating text: the model stores the keys and values it computed for each token, at every layer, so that it does not recompute them for every new token (Attention Is All You Need defines the queries, keys and values; The KV Cache: Why Long Chats Get Slow and Expensive explains the cache step by step). A model run without a cache still has a context window. What the cache determines is what that window costs when you serve it.
What does a long context cost on the hardware?
Generation has two phases with different bottlenecks. Prefill reads the whole prompt at once: every token attends to every earlier token, so the work grows with the square of the prompt's length, and the phase is limited by compute. FlashAttention made this phase much faster by reorganizing attention to cut reads and writes between the GPU's large high bandwidth memory (HBM) and its small on-chip memory (FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, Dao et al.). Decode then writes one token at a time. Each step does little arithmetic but must read the whole cache, so it is limited by memory bandwidth: as Shazeer put it, incremental inference is slow "due to the memory-bandwidth cost of repeatedly loading the large 'keys' and 'values' tensors" (Fast Transformer Decoding: One Write-Head is All You Need, Shazeer 2019). In practice, a long prompt delays the first token, and a large cache slows every token after it.
The cache's size grows in step with the sequence length. For a model with layers, key and value heads of size , stored at bytes per number, one sequence of tokens needs:
The leading 2 counts keys and values. Take Llama 3.1 70B, which has 80 layers and 8 key and value heads of size 128 (its 64 query heads share them), stored at 16 bits, 2 bytes per number (The Llama 3 Herd of Models, Table 3). Each token needs bytes, and a full 128K window (131,072 tokens) needs about 43 GB, or 40 GiB, for a single conversation.
Now put it on GPUs. The weights alone, about 140 GB, fill most of two 80 GB GPUs and leave roughly 20 GB, before any working memory: not enough for even one full 128K token cache. Every long conversation needs about half of another GPU. That is why serving throughput at long contexts is usually limited by KV cache memory, not by compute: the number of requests you can batch together is set by how many caches fit (Efficient Memory Management for Large Language Model Serving with PagedAttention, Kwon et al.). Raise the context length on the same hardware and the batch size, and with it throughput, has to fall.
How do serving systems stretch the limits?
Four techniques do most of the work. The KV Cache: Why Long Chats Get Slow and Expensive and Batching and PagedAttention: How vLLM Serves Hundreds at Once cover them in depth; here is what each buys.
Paged attention stores the cache in fixed size pages, like virtual memory, instead of one contiguous block per sequence reserved at the maximum length. The vLLM paper reports "near-zero waste in KV cache memory", sharing of cache pages between requests with a common prefix, and two to four times the throughput of earlier serving systems at the same latency (Kwon et al.).
Grouped-query attention (GQA) lets several query heads share one key and value head, which shrinks in the formula. Its authors found that it reaches "quality close to multi-head attention" at speeds close to sharing a single key and value head (GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, Ainslie et al.). Llama 3.1 70B's 8 shared heads make its cache eight times smaller than 64 separate ones would.
KV cache quantization stores keys and values in fewer bits. KVQuant reported less than 0.1 perplexity degradation at 3 bits, enough to serve LLaMA-7B with a context of up to 1 million tokens on a single 80 GB A100 (KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization, Hooper et al.).
Offloading keeps the full cache in CPU memory and brings back only what each step needs. The catch is bandwidth: the link between CPU and GPU is far slower than HBM, so research systems such as A²ATS keep a compressed index on the GPU and fetch only the entries likely to matter (A²ATS: Retrieval-Based KV Cache Reduction via Windowed Rotary Position Embedding and Query-Aware Vector Quantization, He et al. 2025).
How does a model know where a token is?
Attention on its own does not know order, so models add position information, and the scheme they use shapes how long inputs behave.
Many current open models, Llama 3 among them, use rotary position embeddings (RoPE). RoPE does not add anything to the token's embedding. Inside each attention layer, it rotates the query and key vectors by angles that depend on their positions, so their dot product depends on how far apart two tokens are. Its authors list "decaying inter-token dependency with increasing relative distances" among its properties: all else equal, nearby tokens interact more strongly (RoFormer: Enhanced Transformer with Rotary Position Embedding, Su et al.). The rotation speeds are set by a base frequency, and model makers raise it for long windows; Meta set Llama 3's to 500,000 "to better support longer contexts".
ALiBi takes a different route. It adds no position embeddings at all and instead "biases query-key attention scores with a penalty that is proportional to their distance." A model trained on 1,024 token inputs this way worked on 2,048 token inputs, matching a model trained at that length (Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation, Press et al.).
Going past the training length is where things break. Simply feeding a RoPE model longer inputs than it saw "may lead to catastrophically high attention scores that completely ruin the self-attention mechanism." Position Interpolation avoids this by scaling the position numbers down to fit the original range, and extended LLaMA models to windows of up to 32,768 tokens with at most 1,000 steps of fine-tuning (Extending Context Window of Large Language Models via Positional Interpolation, Chen et al.). YaRN refines the idea and reports needing 10 times fewer tokens and 2.5 times fewer training steps than earlier methods (YaRN: Efficient Context Window Extension of Large Language Models, Peng et al.). These methods make a longer window usable. They do not make every position in it equally well used.
What is lost in the middle, and what causes it?
Lost in the middle is the finding that models often use information at the start and end of a long input more reliably than information in the middle (Lost in the Middle: How Language Models Use Long Contexts, Liu et al.). In the study's question answering test with 20 documents, one of which held the answer, here is how often three models answered correctly with the answer in the first, the tenth and the last document (from the paper's Table 6):
| Model (2023) | Answer in 1st document | In 10th | In 20th |
|---|---|---|---|
| GPT-3.5-Turbo | 75.8% | 53.8% | 63.2% |
| Claude-1.3 | 59.9% | 56.8% | 60.1% |
| MPT-30B-Instruct | 53.7% | 52.2% | 56.3% |
The size of the effect varies a lot. GPT-3.5-Turbo dropped 22 points from first to middle, below the 56.1% it scored with no documents at all; Claude-1.3's curve was nearly flat. MPT-30B-Instruct uses ALiBi, not RoPE, and still showed a small dip, so RoPE alone cannot be the cause. The same paper found the U shape only in larger Llama 2 models: the 7B models favored only the end of the input, while the 13B and 70B models favored both ends.
What causes it is still being studied, and several things seem to contribute. Positional encodings favor nearby tokens. Causal attention means that tokens early in the input can be read by everything after them. Models also learn to put a lot of attention on the very first tokens whatever they contain, a pattern called attention sinks (Efficient Streaming Language Models with Attention Sinks, Xiao et al.). And training data shapes which positions matter. Hsieh et al. linked lost in the middle to a U-shaped attention bias, in which the first and last tokens "receive higher attention, regardless of their relevance," and showed that calibrating that bias away improved how well models used the middle (Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization).
So lost in the middle is a widely observed tendency, not a law of physics. It differs by model, training and task, and it can be reduced. The practical rule is to put the information and instructions that matter most near the start or end of a long input, and to test your own model at the lengths you actually use.
How much of the window can a model actually use?
The advertised window is the most a model accepts. The effective window is how far it can go while quality holds, and the gap can be large. The simple "needle in a haystack" test, which hides one fact in a long text and asks for it, flatters models because the question often shares words with the needle.
Harder benchmarks show the gap. RULER tested 17 long context models on 13 tasks, including tracing chains of references and aggregating information. Despite nearly perfect scores on the plain needle test, "almost all models exhibit large performance drops as the context length increases," and although all claimed windows of 32K tokens or more, only half kept satisfactory performance at 32K (RULER: What's the Real Context Size of Your Long-Context Language Models?, Hsieh et al.). NoLiMa removed the shared words between question and needle, so models had to make an inference to find it. Of 13 models that claimed at least 128K tokens, 11 fell below half of their short input scores at just 32K (NoLiMa: Long-Context Evaluation Beyond Literal Matching, Modarressi et al.).
Vendors now ship million token windows, and newer models may do better than the ones these studies tested, but the lesson holds: measure on tasks like yours, at your lengths, with questions that do not simply repeat the words of the answer. For ways to work within the effective window (chunking, summarizing, retrieval), see Context Window Management for Big Jobs.
Quick Reference
| Property | Value |
|---|---|
| Context window | The maximum tokens a model can attend over in one pass, set mainly by training |
| KV cache | Stored keys and values per token per layer; sets the memory cost of a long context when serving |
| KV cache size | bytes per sequence |
| Llama 3.1 70B example | 327,680 bytes per token; about 43 GB for 128K tokens at 16 bits |
| Prefill | Grows with the square of the prompt length; limited by compute |
| Decode | Reads the whole cache for every new token; limited by memory bandwidth |
| RoPE distance effect | Attention tends to weaken with distance (not uniform) |
| Lost in the middle | U-shaped recall by position (empirical; size varies by model) |
| Effective vs advertised length | Often much shorter (RULER: half of 17 models satisfactory at 32K) |
| GQA memory reduction vs full multi-head | Query heads divided by KV heads (8 times for Llama 3.1 70B) |
Frequently Asked Questions
Q: If I have a 128K context window, can I just dump 128K tokens of documents in and expect the model to use all of them?
No. The model can attend to every token, but information in the middle of a long input is often used less reliably than information at the start or end, and many models lose accuracy well before their advertised limit. How much depends on the model, so test yours at the length you plan to use, and send only what the task needs.
Q: Does lost in the middle affect structured prompts differently than free text?
The same positional tendency applies. If you place instructions at both the beginning and end of a long prompt with documents in between, the instructions sit in the strongest positions. Documents in the middle are the most likely to be underused, so put the most relevant ones first or last.
Q: Can I fix lost in the middle by repeating important information?
Partially. A second copy near the end puts the information in a strong position, which is why repeating key instructions just before the question helps. A better strategy is to keep inputs short and focused, order documents by relevance, and put critical information near the beginning or end.
Q: Why does my throughput collapse when I increase context length?
Because the KV cache for each request grows in step with its length. With a fixed amount of GPU memory, larger caches mean fewer requests fit in a batch, so fewer tokens per second come out of the same hardware. Without paged attention or prefix sharing, fragmentation and worst case preallocation waste even more memory.
Q: Is ALiBi better than RoPE for long contexts?
ALiBi was designed to extrapolate, and a model trained on 1,024 tokens worked at 2,048. Many current open models, such as Llama 3, use RoPE instead and reach long windows with long context training and extension methods such as Position Interpolation or YaRN. Either way, the choice is made when the model is built; you cannot change it at inference time.
Test yourself
You serve a model with a 128K context window. When users start pasting long documents, throughput halves and the time to the first token jumps. Explain both effects using the KV cache formula, and name two serving changes that would help.
Answer: The cache grows in step with the tokens in each request: bytes. When a request's length goes from a few thousand tokens to tens of thousands, its cache grows by the same factor, so far fewer requests fit in GPU memory at once. The batch shrinks and throughput falls, and each decode step also has to read a much larger cache, which is limited by memory bandwidth. The first token arrives later because prefill must process the whole long prompt, and that work grows with the square of its length. Two changes help. Paged attention removes the memory wasted by reserving maximum length blocks and lets requests share common prefixes. Quantizing the KV cache to 8 or even 3 bits fits several times more tokens in the same memory. A model with fewer key and value heads (GQA) helps too, and at the application level, retrieving only the relevant passages keeps small in the first place.
If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.
Sources
- The Llama 3 Herd of Models (Meta 2024)
- Attention Is All You Need (Vaswani et al. 2017)
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (Dao et al. 2022)
- Fast Transformer Decoding: One Write-Head is All You Need (Shazeer 2019)
- Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al. 2023)
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (Ainslie et al. 2023)
- KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (Hooper et al. 2024)
- A²ATS: Retrieval-Based KV Cache Reduction via Windowed Rotary Position Embedding and Query-Aware Vector Quantization (He et al. 2025)
- RoFormer: Enhanced Transformer with Rotary Position Embedding (Su et al. 2021)
- Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (Press et al. 2021)
- Extending Context Window of Large Language Models via Positional Interpolation (Chen et al. 2023)
- YaRN: Efficient Context Window Extension of Large Language Models (Peng et al. 2023)
- Lost in the Middle: How Language Models Use Long Contexts (Liu et al. 2023)
- Efficient Streaming Language Models with Attention Sinks (Xiao et al. 2023)
- Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization (Hsieh et al. 2024)
- RULER: What's the Real Context Size of Your Long-Context Language Models? (Hsieh et al. 2024)
- NoLiMa: Long-Context Evaluation Beyond Literal Matching (Modarressi et al. 2025)
