Why RAG Exists: The Context Problem

A language model does not know your company handbook. It cannot. Its weights were fixed when training ended, so every question you ask today is answered from a snapshot of public text gathered months or years ago. Retrieval-Augmented Generation (RAG) is the architecture that bridges this gap. Before the model answers, a search step finds the few passages of your own documents that matter for this question and puts them into the prompt. The model then answers from those passages instead of from memory.

You might ask why we need a search step at all, now that the newest models accept about a million tokens. That is a fair question, and sometimes the honest answer is that you do not need RAG. But a long prompt is not read with equal care from start to finish, every token you send costs time and money, and most real document collections change, grow and need answers that point to their source. This article explains the problem RAG solves, when it is worth the extra machinery, and what the rest of this series builds.

Why can't you just paste all your documents into the prompt?

Think of the model's context window as a big whiteboard. Everything you want the model to use has to be written on it, and these days the whiteboard is very large. As of October 2026, the newest models from OpenAI, Anthropic and Google each accept about a million tokens (Anthropic's models overview, OpenAI's GPT-6 Astra model page, Google's Gemini models page). The catch is that the model does not read every part of the whiteboard equally well.

Now imagine you are building a "chat with your company handbook" assistant. The handbook is 200 pages of policies, leave rules and expense guidelines. At a typical few hundred words a page, that is very roughly 80,000 words, or somewhere around 100,000 to 150,000 tokens depending on the tokenizer. It fits on the whiteboard with room to spare. So why not paste it in and ask "How many vacation days does a new hire get after six months?"

Because where a fact sits in a long prompt changes how reliably the model uses it. The best known study is Lost in the Middle (Liu et al. 2023, Lost in the Middle: How Language Models Use Long Contexts). The researchers gave models with windows from 4,000 to 100,000 tokens a question plus 10, 20 or 30 documents, only one of which held the answer, and moved that document around. Accuracy was usually highest when the answer sat at the start or the end of the prompt, and lowest when it sat in the middle: a U-shaped curve. These prompts were only a few thousand tokens long, so the problem appears far below the advertised limit. The authors call longer prompts a trade-off: more text may contain the answer, but it also gives the model more to reason over, which can lower accuracy. In their open-domain question answering test, using more than 20 retrieved documents improved answers only marginally.

Advertised size and useful size are different things. The RULER benchmark tested 17 models that all claimed windows of 32,000 tokens or more, and found that only half of them still performed satisfactorily at 32,000 (Hsieh et al. 2024, RULER: What's the Real Context Size of Your Long-Context Language Models?). Newer models do better on these tests, and the gap depends on the task, so the honest rule is to measure on your own questions rather than trust the number on the model page. Context Windows: The Model's Working Memory covers this in more depth.

Why does the middle get less attention? Researchers are still working that out. One study traced it to an attention bias inside the models themselves: tokens at the start and the end of the input receive more attention regardless of whether they are relevant (Hsieh et al. 2024, Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization). How a model encodes position, and where important text tends to sit in its training documents, are other likely parts of the story.

Then there is cost. You pay for every token you send, in money and in waiting time, on every question. Pasting a 120,000 token handbook into each request means paying for 120,000 tokens to answer "Is Friday a remote day?"

When is a long prompt the better choice?

Sometimes it is. Prompt caching changes the arithmetic: if every request starts with the same long text, the provider can reuse its work on that prefix and bill it at a fraction of the normal price. As of October 2026, cached input on the newest models costs a tenth of the normal input price or less at both Anthropic and OpenAI, and OpenAI applies caching automatically (Anthropic's prompt caching guide, OpenAI's prompt caching guide).

So if your documents are small enough to fit, change rarely, and you can live with the position effects above, putting them in a cached prompt is a reasonable design with fewer moving parts than RAG. It also helps when a question needs the whole document at once, such as "summarize every change in this handbook since last year".

RAG earns its extra machinery when one or more of these is true:

• The collection is bigger than any window. A company wiki, years of support tickets or every product manual will not fit, cached or not.
• The content changes often. Updating a search index takes minutes; you only re-process the documents that changed.
• You need citations. Retrieval knows exactly which passage each answer came from, so the answer can point to it (Part 6 builds this).
• Different people may see different documents. A search step can filter out what a user is not allowed to read before the model ever sees it (Part 4 shows how a vector database does this).
• Cost and speed per question matter at scale. Sending a few relevant passages is cheaper and faster than sending everything, even with caching.

Why not fine-tune the model on the handbook instead?

Every few months someone asks this. Fine-tuning means continuing to train the model on your own examples so that its weights change. It is good at changing how a model behaves, such as its tone or output format, and much less reliable as a way to store facts that change: you need training data, a new training run every time the handbook changes, and there is still no guarantee the model recalls the exact figure when someone phrases the question differently. Fine-Tuning, Explained: Teaching an Old Model New Tricks covers how it works, and Fine-Tuning vs RAG vs Prompting: Which Lever When shows how to choose between the three.

The original RAG paper framed the choice as two kinds of memory (Lewis et al. 2020, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks). Parametric memory is the knowledge stored in the model's weights. Non-parametric memory is an external index the model can look things up in; in the paper it was a vector index of Wikipedia searched by a neural retriever. You update an index by re-processing the documents that changed, not by retraining the model. The model stays the same, and the index becomes the source of truth.

How does RAG work at a high level?

RAG does not make the model smarter. It builds a pipeline that picks, for each question, the small set of passages most likely to contain the answer, and hands only those to the model.

Context stuffing vs RAG

Whole handbook in the prompt

• Roughly 100,000 tokens or more on every question
• Facts in the middle are used less reliably
• Every question pays for every token, less with caching
• Few moving parts

RAG

• Only the best matching chunks, a few thousand tokens
• The model reads a short, focused context
• Each answer can cite its passage
• More moving parts, each able to fail

Here is the flow for the handbook assistant. Ahead of time, the handbook is split into short passages called chunks, and each chunk is turned into an embedding: a list of numbers produced by an embedding model, arranged so that texts with similar meaning get similar numbers (you met these in Embeddings: Meaning as Numbers). The embeddings go into a vector database, which can quickly find the stored vectors closest to a new one.

When a user asks "What is the expense limit for client dinners?", the system turns the question into an embedding the same way, finds the top-k chunks, meaning the k chunks whose embeddings are closest, and optionally re-orders them with a more careful model. It then packs those chunks into the prompt with the question, and the model writes an answer based on them.

Diagram loading

The quality comes from the retrieval steps, not from the model. The chunking strategy, the embedding model, the search and the re-ranker together decide which handful of passages the model will see. If retrieval picks the wrong passages, the model never sees the right information. If it picks too many, the important passage can end up in the middle of a long prompt, where we saw models use it less reliably.

This is why people who build RAG systems spend so much time on chunking, embeddings and hybrid search. A bad chunking strategy splits a policy across two chunks, and the search never returns the complete rule. A search that only compares meanings can miss an exact code such as "expense code 45B", because the embedding model has no special sense of what that string means. These are not model failures. They are retrieval failures, and each part of this series takes one of them apart.

What will you build in this series?

From Part 2 on, each part adds one small Python file to a working handbook assistant that runs on a laptop, with free local models and no paid API calls:

• Part 2, Chunking: The Decision Everything Downstream Inherits, splits a sample handbook into chunks.
• Part 3, Embeddings and Vector Search, Demystified, turns the chunks into embeddings and searches them.
• Part 4, Vector Databases: What Actually Matters When Choosing, stores them in a vector database with permission filters.
• Part 5, Hybrid Search and Re-Ranking: The Cheapest Quality Win, adds keyword search, fusion and a re-ranker.
• Part 6, Grounding and Citations: Answers You Can Check, builds a prompt with citation IDs and checks every citation the model gives.
• Part 7, Evaluating RAG: Retrieval Metrics That Predict Answer Quality, measures retrieval quality and fails a build when it gets worse.
• Part 8, Agentic RAG: When the Retriever Starts Thinking, looks at agents that decide for themselves what to search for.

Quick Reference

Property	Value
Context window of the newest models	About 1 million tokens at OpenAI, Anthropic and Google (as of October 2026)
Where facts are used best	Often the start and end of a long prompt; worst in the middle (Lost in the Middle)
Advertised vs useful length	In RULER, only half of 17 models claiming 32K or more stayed satisfactory at 32K
What you pay for	Every token you send, on every request (cached prefixes cost less)
When to skip RAG	Small, stable documents that fit the window; consider prompt caching
When RAG wins	Large or changing collections, citations, per-user permissions, cost per question
Key architectural split	Parametric memory (weights) vs non-parametric memory (an index)

Frequently Asked Questions

Q: If a model accepts a million tokens, why not put everything in the prompt and let it figure it out? For a small, stable set of documents, that can be the right call, especially with prompt caching. But models use information in the middle of a long prompt less reliably than information at the start or end, every question pays for every token, and nothing in the prompt tells you which passage an answer came from. For large, changing or permissioned collections, retrieval does the job better.

Q: Can't I just fine-tune the model on my company data instead of using RAG? Fine-tuning is better at changing how a model behaves than at storing facts that change. Every handbook update would need a new training run, and the model may still not recall the exact rule. Most teams use retrieval for facts and keep fine-tuning, if they use it at all, for behavior.

Q: What is the difference between the maximum context window and the effective one? The maximum is the hard limit the API enforces; a longer request is refused. The effective length is how much text the model can use while quality stays acceptable for your task. Benchmarks such as RULER show it can be much lower than the maximum, and it varies by model and task.

Q: Does RAG completely solve the hallucination problem? No. RAG reduces made-up answers by giving the model the right text to work from. But if retrieval picks the wrong chunk, or the model ignores or misreads the text it was given, the answer can still be wrong. That is why Part 6 checks every citation.

Q: Why is the chunking strategy so important if the model only sees a few chunks anyway? Because the chunks are the only units the search can return. If a rule was split badly, no chunk contains it whole, and the model never gets the chance to answer correctly. Chunking decides what retrieval can ever find.

Test yourself

Your handbook assistant splits the handbook into fixed 512-token chunks, cutting wherever the count runs out, with no overlap. The handbook says: "Employees may claim client entertainment up to $150 per person, but advance approval is required for any expense above $75." A user asks: "What is the client entertainment expense limit?" The search returns a chunk that begins in the middle of that sentence: "$75. Use expense code 45B for client meals." The assistant answers: "The limit is $75." What went wrong, and how would you fix it?

Answer: The fixed-size chunker cut the policy mid-sentence. The limit of $150 per person and the rule that spending above $75 needs advance approval ended up in the previous chunk, which the search did not return. The model saw only "$75" next to the question and took it for the limit. The correct answer is up to $150 per person, with advance approval for anything above $75. To fix it, split on the document's structure so that each policy stays whole (Part 2 shows how), and never cut mid-sentence. Overlap between chunks helps only when the repeated text is large enough to carry the whole rule, so check that it does. Another fix is to search small chunks but hand the model the whole section they came from. Citations with quotes (Part 6) would also have exposed the gap, because the quoted text does not state a limit.

Next: Chunking: The Decision Everything Downstream Inherits. It splits a sample handbook into chunks and shows how chunk size, overlap and structure decide what your assistant can find.

If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.

Sources

• Lost in the Middle: How Language Models Use Long Contexts (Liu et al. 2023)
• RULER: What's the Real Context Size of Your Long-Context Language Models? (Hsieh et al. 2024)
• Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization (Hsieh et al. 2024)
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al. 2020)
• Models overview (Anthropic, read October 2026)
• GPT-6 Astra model page (OpenAI, read October 2026)
• Gemini models (Google, read October 2026)
• Prompt caching (Anthropic, read October 2026)
• Prompt caching (OpenAI, read October 2026)