Context Windows: The Model's Working Memory

A context window is the maximum number of tokens a language model can process in one go. It includes the system prompt, conversation history, and the model's own output. Inside the model, self-attention and a key-value cache enforce this limit, acting as the model's working memory. When the window fills up, older information falls out, and the model can no longer use it.

But here is the twist: the model's performance drops even when the window is not full. If you bury a crucial detail in the middle of a long prompt, the model often ignores it. And in a long chat, the assistant might start hallucinating long before you hit the advertised token limit.

In the last episode, we saw how training and inference are separate phases. Now we will look at the inference-time machinery that decides how much of your conversation the model can actually keep in mind.

What exactly is a context window?

Imagine you are reading a long recipe on a small phone screen. You can only see a few lines at a time. To follow the recipe, you scroll up and down, but you cannot see the whole thing at once. The visible area is your context window.

A language model works the same way. It reads text in chunks called tokens, not words. A token is a small piece of text, often a word fragment. The model can only “see” the tokens that fit inside its context window. If the recipe is 10,000 tokens and the window is 4,000, the model will only read the first 4,000 tokens (or the last 4,000 if you truncate from the beginning). It will miss the baking temperature at the end.

Every message you send and every word the model generates consumes tokens from the same finite window. When you ask a chatbot for a recipe, the prompt plus the model's reply might use 500 tokens. If you then ask “what was the first ingredient?”, the model needs the earlier conversation to answer. But if the total token count exceeds the window, the earliest messages get pushed out, and the model literally cannot see them anymore. This is why long chats eventually lose track of what was said at the beginning. OpenAI tokenizer

How does the model “remember” everything within the window?

Think of the model as a reader who takes notes on a scratchpad. Instead of re-reading the whole book each time they write a new sentence, they jot down key points about every previous sentence. That scratchpad is the key-value cache, or KV cache.

When you first send a prompt, the model does a full read, called the prefill phase. It processes every token and stores two vectors for each token at every layer: a key and a value. These vectors capture what the token means and how it relates to others. Once the prefill is done, the model has a complete set of notes for the entire input.

When it generates the next word, it only needs to look at its cached notes and the new word. It does not re-process the whole history. This is why generation after the first token is fast. The KV cache grows with each new token, and it lives in GPU (graphics processing unit) memory. For a model with many layers and heads, the cache can easily become larger than the model weights themselves. Attention Is All You Need

A simplified view of the process looks like this:

# Pseudocode for autoregressive decoding with KV cache (simplified)
tokens = tokenize(prompt)
logits, kv_cache = model.forward_prefill(tokens)
next_token = sample_from_logits(logits[-1])
generated = [next_token]

for _ in range(max_new_tokens - 1):
    logits, kv_cache = model.forward_decode(next_token, kv_cache)
    next_token = sample_from_logits(logits)
    generated.append(next_token)

text = detokenize(generated)

Why does the context window have a hard limit?

The attention mechanism is the reason. Attention lets every token look at every other token to decide what is important. If you have 4,000 tokens, the model computes 16 million pairwise comparisons per layer. Double the window to 8,000 tokens, and you quadruple the comparisons. This quadratic growth quickly becomes too expensive in time and memory.

Even with the KV cache, each new token still attends to all previous tokens. So the cost per generated token is linear in the sequence length, and the memory for the cache grows linearly too. A 128,000-token context can require hundreds of gigabytes of GPU memory just for the cache. That is why every model has a maximum sequence length baked into its architecture. Attention Is All You Need

In the recipe analogy, if you had to compare every ingredient with every other ingredient to understand the recipe, a longer recipe would become impossibly slow. The context window is the model's way of saying “I can only handle this many comparisons at once.”

Why do long conversations get weird?

Two phenomena cause trouble: the “lost in the middle” effect and the mismatch between advertised and training context lengths.

Models pay the most attention to the beginning and the end of the context. If you put a crucial instruction in the middle of a long prompt, the model often ignores it. In a chat, a follow-up question that relies on something said 20 turns ago (now sitting in the middle of the window) may get a wrong answer. The model simply does not give that middle region the same weight. Lost in the Middle

The second issue is that the model's training context length is often smaller than the API (application programming interface)'s maximum. Llama 3, for example, was trained on 8,192-token sequences. When you push it to 32,000 tokens, the positional encodings that tell the model the order of words become unreliable. The model was never taught to handle positions that far out, so its predictions degrade. This happens even if the GPU has plenty of memory. Llama 3 technical report

In stateful multi-turn apps that reuse the KV cache across turns, the cache grows monotonically. Once it exceeds the training context length, quality drops sharply. The assistant might start repeating itself, contradicting earlier statements, or inventing facts. This is why long conversations get weird long before you hit the advertised token limit.

How do systems try to extend or manage the context window?

The simplest method is truncation: keep only the most recent N tokens and drop the rest. This is like scrolling the phone screen so you always see the last few lines. It works, but you lose all earlier context.

A smarter approach is retrieval-augmented generation (RAG). Instead of stuffing the entire knowledge base into the window, you search for relevant documents and put only those into the prompt. This keeps the window small and focused. RAG paper

Some systems try to compress the KV cache itself. They use learned modules that distill a long cache into a fixed-size summary. This is like writing a one-paragraph summary of the recipe instead of keeping every line. However, these methods must be careful with positional encodings. If you delete tokens from the middle of the cache, you break the model's sense of order. Many compression strategies fail because they scramble the relative positions that models like Llama 3 rely on. KV cache compression survey

The most robust strategy today is often the simplest: keep a contiguous block of the most recent conversation and either summarize or discard the rest. This preserves positional integrity, even if it throws away more tokens.

Quick Reference

Frequently Asked Questions

Q: Does a larger context window always mean better performance? No. Models may not be trained to use very long contexts effectively. Attention can get diluted, and retrieval quality often drops as the window grows.

Q: Why does my assistant forget the beginning of the conversation even though it claims 128k context? The app probably truncates the history to save cost or latency. Or the model's effective context is much smaller than the advertised number because of training limits.

Q: How does RAG compare to just using a huge context window? RAG fetches only the relevant information, reducing noise and compute. A huge window forces the model to sift through everything, which can hurt accuracy and increase latency.

Q: What is the difference between architectural context limit and effective context limit? The architectural limit is the maximum number of tokens the model can physically process. The effective limit is where the quality remains acceptable. The effective limit is often much lower.

Q: Can I train a model to have an infinite context window? Not with standard attention. You would need architectural changes like linear attention or external memory. These are still active research areas.

Test yourself

You are debugging a customer support bot that uses a long conversation history. After 50 turns, the bot starts giving irrelevant answers. What is likely happening, and how would you diagnose it?

Answer: The accumulated KV cache has likely grown beyond the model's architectural context limit. Even if the API says 128k tokens, the model may have been trained on only 8k. Once the cache exceeds that, positional encodings misalign and attention degrades. You can check the total token count of the conversation. Try truncating to the last 20 turns and see if quality improves. Also, look for “lost in the middle”: important context from earlier turns may now sit in the middle of the window and be ignored. Moving that context to the very beginning or end of the prompt can help. If the problem persists, implement a summarization step that compresses older turns into a short gist before the cache grows too large.

If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.

Sources

• OpenAI tokenizer
• Attention Is All You Need
• Lost in the Middle: How Language Models Use Long Contexts
• Llama 3 technical report
• Retrieval-Augmented Generation for Knowledge-Intensive NLP (natural language processing) Tasks
• A Survey on Efficient Inference for Large Language Models
• internalsdecoded.com