Context Windows: The Model's Working Memory
What fits, what falls out, and why long conversations get weird.
A context window is the most tokens a language model can take in at once. Everything counts toward it: the instructions, the conversation so far, any documents you attach, and the reply the model is writing. Anthropic's glossary calls it the model's "working memory" (Anthropic's glossary). When a conversation outgrows the window, something has to give: either the request is refused, or the app drops or summarizes older messages.
The model's performance can drop even when the window is not full. If you bury an important detail in the middle of a long prompt, the model often misses it. And in a long chat, the assistant can lose track of things long before you reach the advertised limit.
In Training vs Inference: Why Building Costs Millions and Asking Costs Cents, you saw that answering a question is a separate phase from training. Now we look at how much of your conversation the model can actually keep in view while it answers.
What exactly is a context window?
Imagine you are reading a long recipe on a small phone screen. You can only see a few lines at a time. The visible area is your context window.
A model works the same way, except that its screen is measured in tokens. As you saw in Tokens: The Currency of AI, the model reads tokens, not words, and it can only use the tokens that fit inside its window. Whatever does not fit is not seen at all. Send a prompt longer than the window through an API and the request is refused; Anthropic's API, for example, answers that the "prompt is too long". Chat apps handle it for you: Anthropic notes that chat interfaces such as its claude.ai can drop the oldest parts of a conversation first (Anthropic's guide to context windows).
Windows have grown enormously. As of October 2026, the newest models from OpenAI, Anthropic and Google each take about a million tokens (OpenAI's GPT-6 Astra model page, Anthropic's models overview, Google's Gemini models page). Anthropic says a million tokens is roughly 555,000 words on its current tokenizer and about 750,000 on its earlier one. An average adult reads English non-fiction at about 238 words a minute (How many words do we read per minute?, Brysbaert 2019), so reading that much would take a person roughly 40 to 50 hours.
Every message you send and every token the model writes uses up the same window. When you ask a chatbot for a recipe, the prompt and the reply might use 500 tokens. If you then ask "what was the first ingredient?", the model needs the earlier messages to answer, and the app sends them back to it with your new question. In a very long conversation, an app that trims from the start drops the earliest messages first, and the model simply cannot see them anymore.
How does the model keep track of everything in the window?
Think of the model as a reader who takes notes on a scratchpad. Instead of rereading the whole book before writing each new sentence, it keeps short notes about every token it has already read. That scratchpad is called the key value cache, or KV cache.
When you send a prompt, the model first reads all of it at once and writes notes for every token at every layer. Then, as it writes each new token, it only has to read its notes and process the one new token, instead of redoing the whole history. That is why replies flow quickly once they start.
The notes cost memory, and they grow with every token. The cache lives in the GPU's memory next to the model itself, and managing it well is one of the hardest parts of serving many users at once (Efficient Memory Management for Large Language Model Serving with PagedAttention). For a large model, one long conversation can need tens of gigabytes of cache. The KV Cache: Why Long Chats Get Slow and Expensive works through the arithmetic.
Why does the context window have a limit?
The limit comes mostly from two things: how long the texts were that the model was trained on, and how much computing and memory it takes to serve long inputs.
Training comes first. A model learns to use positions and distances only as far as its training texts went, so long windows have to be trained on purpose. Meta, for example, first trained Llama 3 on texts of 8,000 tokens, then continued training on longer texts to stretch its window to 128,000 (The Llama 3 Herd of Models).
Cost comes second. Attention lets every token look back at the tokens before it, so the number of pairs grows with the square of the length, as you saw in Attention, Gently: How Models Decide What Matters. Make the input 32 times longer and the pairs grow about a thousand times. The KV cache grows more gently, in step with the length, but at long lengths it takes a lot of memory for every single conversation.
- About 17 million token pairs per layer
- About 1.3 GB of KV cache
- Quick to read in
- About 17 billion token pairs per layer
- About 43 GB of KV cache
- Much slower to read in
In the recipe picture, it is as if you had to compare every line with every earlier line before you could cook. A longer recipe gets slow fast, so model makers pick a window they can train well and afford to serve.
Why do long conversations get weird?
Several things go wrong as the text gets long, and none of them needs the window to be full.
The middle gets lost. Researchers found that models often do best when the information they need sits at the beginning or end of a long input, and much worse when it sits in the middle, "even for explicitly long-context models" (Lost in the Middle: How Language Models Use Long Contexts). In a chat, a fact you mentioned twenty turns ago now sits in that middle.
The advertised size is not the useful size. When the RULER benchmark tested 17 long context models, almost all of them got worse as the input grew, and although every one claimed a window of 32,000 tokens or more, only half kept up satisfactory performance at 32,000 (RULER: What's the Real Context Size of Your Long-Context Language Models?). Anthropic's own guide puts it plainly: "As token count grows, accuracy and recall degrade, a phenomenon known as context rot" (Anthropic's guide to context windows).
The history gets cluttered. A long chat collects corrections, changed plans and abandoned ideas. The model sees all of them at once, so an instruction you replaced an hour ago is still sitting there beside the new one.
The app quietly trims. To save cost and room, many apps drop or summarize older turns. If an app trims from the start, your first instructions are the first to go, and the model is then answering without them.
That is why long conversations can turn odd, repetitive or forgetful well before any limit is reached.
When you want to go deeper, Context Windows Are Not What You Think picks up from here: what sets the limit, what a long context costs on the hardware, and why the number on the label is not the length a model can actually use.
How do systems manage long contexts?
The simplest method is truncation: keep only the most recent part of the conversation and drop the rest. It is like scrolling the phone screen so you always see the last few lines. It works, but everything earlier is lost.
A step up is summarizing. The app replaces old turns with a short summary and keeps the recent ones whole. Anthropic offers this on its API as compaction, which "automatically summarizes earlier parts of the conversation on the server, so the conversation can continue past the context window limit" (Anthropic's guide to context windows).
Another approach is retrieval augmented generation, or RAG. Instead of stuffing a whole knowledge base or history into the window, the system searches it for the passages that matter to the current question and puts only those into the prompt (Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks). That keeps the window small and focused.
Researchers are also working on shrinking the KV cache itself, by storing it more compactly or keeping only the most useful notes. These methods trade some accuracy for memory and are still an active research area (A Survey on Efficient Inference for Large Language Models).
A dependable approach is often the simplest: put the key facts and instructions at the start, keep the recent conversation whole, and summarize or retrieve the rest.
Quick Reference
| Property | Value |
|---|---|
| What counts toward the window | Instructions, conversation history, attachments and the reply being written |
| Newest models' window | About 1 million tokens at OpenAI, Anthropic and Google (as of October 2026) |
| Other common windows | 128K tokens for many open models, such as Llama 3.1 |
| Words per million tokens | About 555,000 to 750,000, depending on the tokenizer (Anthropic's figures) |
| Over the limit | APIs refuse the request; chat apps trim or summarize older turns |
| KV cache | The model's notes on every token so far; grows with every token, tens of gigabytes for one long conversation on a large model |
| Lost in the middle | Details in the middle of a long input get used less than those at the start or end |
Frequently Asked Questions
Q: Does a larger context window always mean better performance? No. A bigger window lets more fit, but models tend to use long inputs less reliably, and accuracy and recall fall as the token count grows. Sending only what matters usually beats sending everything.
Q: Why does my assistant forget the beginning of the conversation when it claims a huge window? Usually for one of three reasons. The app may trim or summarize old messages to save cost. The detail may now sit in the middle of a long history, where models look least carefully. Or the model's useful range may be shorter than its advertised one.
Q: How does RAG compare to just using a huge context window? RAG fetches only the relevant information, which cuts noise, cost and waiting time. A huge window makes the model sift through everything, which can hurt accuracy. Many systems use both: a large window, filled carefully.
Q: What is the difference between the architectural limit and the effective limit? The architectural limit is the most tokens the model can accept. The effective limit is how far it can go while the quality stays acceptable. The effective limit is often much lower.
Q: Can a model have an unlimited context window? Not with standard attention, because the cost keeps climbing with length. Windows keep growing, and Meta advertises 10 million tokens for its Llama 4 Scout model (Llama 4 model card), but unlimited windows would need different designs, such as external memory, which are still being researched.
Test yourself
You are debugging a customer support bot that keeps a long conversation history. After about 50 turns, the bot starts giving answers that ignore what the customer said earlier and even break its own rules. What is likely happening, and how would you check and fix it?
Answer: First, check the token count: 50 turns is usually far below any current model's window, so the window is probably not full. Next, check whether the app trims or summarizes old turns, and whether the original instructions survive that step; if it trims from the start, the rules may simply be gone. Then look for lost in the middle: the customer's earlier details now sit in the middle of a long history, where models use information least reliably, and the history may be cluttered with corrections. To fix it, pin the key instructions and facts at the start of every request, summarize older turns into a short note, and retrieve only the parts of the history that matter to the current question instead of sending all of it.
Next: Why Models Make Things Up. It explains why a model sometimes answers with confidence and gets it wrong, and what you can do about it.
If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.
Sources
- Glossary (Anthropic)
- Context windows (Anthropic, read October 2026)
- Models overview (Anthropic, read October 2026)
- GPT-6 Astra model page (OpenAI, read October 2026)
- Gemini models (Google, read October 2026)
- How many words do we read per minute? A review and meta-analysis of reading rate (Brysbaert 2019)
- Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al. 2023)
- The Llama 3 Herd of Models (Meta 2024)
- Lost in the Middle: How Language Models Use Long Contexts (Liu et al. 2023)
- RULER: What's the Real Context Size of Your Long-Context Language Models? (Hsieh et al. 2024)
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al. 2020)
- A Survey on Efficient Inference for Large Language Models (Zhou et al. 2024)
- Llama 4 model card (Meta)
