Chunking: The Decision Everything Downstream Inherits
Sizes, overlap, and semantic boundaries. Get this wrong and nothing after it can save you.
Chunking is the step where raw documents become the units your retriever searches and your LLM (large language model) reads. Chunk boundaries decide which facts stay together, and nothing downstream can see across a boundary drawn in the wrong place. Chunking even changes what your retrieval metrics measure; more on that in the evaluation section.
In this part you will also build the first file of the series' handbook assistant: a script that splits a sample handbook into chunks, measured in the same tokens the embedding model will read.
How does chunking fit into the RAG pipeline?
Chunking runs during indexing, after document parsing and before embedding. You take a parsed document, cut it into pieces, and each piece becomes the unit you embed, store and later retrieve. The retriever never sees whole documents. It sees chunks. The LLM never sees whole documents either. It sees whatever chunks the retriever pulled, stitched into a prompt.
This means chunking sets the grain of your whole knowledge index. If a critical fact only ever appears inside a long chunk full of unrelated text, retrieval becomes a lottery. A long chunk's embedding reflects the broad meaning of everything in it, so one buried fact gets diluted and the chunk may not land near a question about that fact (Pinecone's chunking guide).
Formally, a document is a sequence of tokens. A chunking strategy maps that sequence to a set of contiguous segments, each below a maximum size. Each segment gets embedded on its own. At query time, the user's question is embedded, and the system finds the chunks whose vectors are closest to the question's vector. The top-k chunks, plus their text and metadata, become the context the LLM reads.
The pipeline knows nothing about relationships between chunks unless you encode them. If answering a question needs facts from chunks 3, 7 and 12, the retriever must surface all three. If it surfaces only two, the LLM works with incomplete evidence. If it surfaces twenty, the important ones may be buried among the rest and token costs grow. Chunking shapes this tradeoff before any retrieval logic runs.
Why can't we just use whole documents?
Two hard constraints make whole-document retrieval impractical for most real collections. First, embedding models have fixed input limits. As of October 2026, all-MiniLM-L6-v2 truncates anything longer than 256 word pieces, bge-small-en-v1.5 reads up to 512 tokens, OpenAI's text-embedding-3 models accept 8,192, and some newer models such as the Qwen3-Embedding family accept 32K (the all-MiniLM-L6-v2 model card, the bge-small-en-v1.5 model card, OpenAI's embeddings guide, the Qwen3-Embedding model card). A single manual or report can run to hundreds of thousands of tokens.
Second, even with a long input limit, one vector for a whole document says little about any single fact in it. A 200-page document covers hundreds of topics, and its embedding blurs them together. The retriever would rank documents by general subject, not by whether they contain the answer.
There is a third, practical constraint: cost. Embedding and LLM API (application programming interface) calls are priced per token. If you send 50,000 tokens of context with every question when only 500 are relevant, you pay for the other 49,500 every time. Chunking lets you retrieve only the relevant slices.
So the question is not whether to chunk. It is how.
What happens when chunk size is too small?
Small chunks give you precision. A 128-token chunk might contain exactly one definition, one instruction or one data point. When a question matches it, the retrieved text is focused.
But small chunks also cut context away. Imagine your company handbook states: "Employees based in California receive an additional 24 hours of sick leave per year. This policy does not apply to contractors." If a chunk boundary falls between those two sentences, the retriever might surface only the first. The LLM then tells a contractor they get extra sick leave, because the chunking hid the exception.
Small chunks also force you to retrieve more of them to cover the same ground. If five small chunks hold what one well-sized chunk would have, there are more slots for irrelevant chunks to sneak into the top-k, more tokens in the prompt, and more chances that one of the pieces a rule needs is simply not retrieved.
In NVIDIA's 2025 chunking benchmark, very small chunks of 128 tokens and very large ones of 2,048 tokens generally did worse than medium sizes, and 128 tokens was the worst setting on one of the five datasets (NVIDIA's chunking benchmark).
What happens when chunk size is too large?
Large chunks keep context together. A 2,048-token chunk might hold an entire policy section, with its definitions, conditions and exceptions. When the retriever finds it, the LLM has everything it needs.
The problem is that large chunks dilute the embedding. A chunk that covers five topics gets a vector that blends all five. A question about one of them might rank that chunk below a smaller, more focused chunk from a less relevant document.
Large chunks also waste tokens. If your retriever pulls three 2,000-token chunks to answer a question that needs 300 tokens of context, you pay to process thousands of tokens of irrelevant text, on every question. And text that looks relevant but is not can pull the model toward the wrong answer.
The sample handbook
To make this concrete, here is the small sample handbook the rest of this series uses. Save it as handbook.md. It is short so the examples stay readable, but it is shaped like a real one: numbered sections, conditions, exceptions and a couple of codes.
How does recursive character splitting actually work?
Recursive character splitting is LangChain's recommended splitter for generic text and a common starting point for RAG systems (LangChain's recursive splitter guide). It tries to keep paragraphs and sentences intact, and only splits at a lower level when a piece is still too large.
The algorithm takes a list of separators, from the biggest unit to the smallest. The default list is double newlines (paragraph breaks), single newlines (line breaks), spaces (word boundaries) and finally the empty string (single characters). Given a text and a target size, it first splits on double newlines. Any piece that is still larger than the target is split again with the next separator, and so on. Only when it runs out of separators does it fall back to cutting between characters.
Then it merges adjacent pieces greedily: it keeps adding pieces to the current chunk until the next one would push it over the target size, then emits the chunk and starts a new one. The overlap is a configurable amount, measured by your length function, carried over from the end of the previous chunk. One detail matters: the overlap is built from whole pieces. If the last piece of a chunk is a paragraph longer than the overlap, nothing is carried over at all.
Here it is on the sample handbook. Chunk sizes should be measured in the tokens your embedding model reads, so the length function uses the tokenizer of the model Part 3 will embed with. The size here is tiny on purpose, so a toy handbook shows what happens to a long section in a real one.
When we ran it (langchain-text-splitters 1.1.3, October 2026), the vacation policy came out as two chunks:
The split landed at the paragraph break, which is what the splitter is designed to do. But it separated the 90-day waiting period from the accrual rate, and the second chunk does not even say it is about vacation in its first line. Notice also that the 20-token overlap repeated nothing: the last piece of the first chunk was a whole paragraph, longer than 20 tokens. A real handbook section is much longer than this one, and a 512-token boundary can fall anywhere inside it in exactly the same way.
How well does recursive splitting do in practice? Chroma's chunking study tested LangChain's recursive splitter at 200, 400 and 800 tokens, with no overlap or with 50 percent overlap, retrieving five chunks per question with OpenAI's text-embedding-3-large (Chroma's chunking evaluation). Average recall, the share of the needed text that the five chunks contained, was about 88 to 90 percent at 200 and 400 tokens. Overlap did not help: at 400 tokens, recall was 89.5 percent without overlap and 88.1 percent with 200 tokens of overlap. Results varied a lot by corpus; at 400 tokens without overlap, recall ranged from about 84 percent on medical papers and financial reports to 100 percent on chat logs. So recursive splitting is a sensible default, and your own documents decide the rest.
When should you use structure-aware chunking instead?
Structure-aware chunking uses the document's own organization to draw the boundaries. If your documents are Markdown with clear headings, split at the headings. If they are PDFs with meaningful pages, split at pages. If they are HTML with semantic tags, split at sections.
This works because authors already grouped related content together. A subsection of a manual usually covers one topic. By following those boundaries, you get coherent chunks without an embedding model having to detect topic shifts. In NVIDIA's benchmark, page-level chunking had the highest average accuracy of the strategies tested, 0.648, and the most consistent results across datasets, and NVIDIA suggests starting with it as the default.
For our handbook assistant, structure is the obvious choice. Handbooks are organized into sections with clear headings, and each section is a natural chunk. LangChain's Markdown splitter splits on headings and keeps each section's headings as metadata. It does not limit the size of a section, so you run the recursive splitter afterwards on anything still too long (LangChain's Markdown header splitter guide). This is the series' first file, chunking.py:
Running python chunking.py printed:
Every policy stays whole, the vacation section included, and every chunk gets a stable ID taken from its section number. Part 6 uses those IDs as citations. The splitter removes the heading lines from the text by default, so the script puts the heading path back at the front of each chunk: "Time off > Vacation." That way a chunk about vacation says so, even if its own sentences only say "accrue 1 day per month". In Part 5, on hybrid search and re-ranking, you will see this prefix rescue a question that never uses the word "vacation".
What is semantic chunking and when is it worth the cost?
Semantic chunking uses embedding similarity to decide where topics change. You split the document into sentences, embed each one, then compare each sentence with the next. When the similarity drops below a threshold, you have found a topic boundary, and you group the sentences between boundaries into chunks, within size limits.
The intuition is clean: if two neighboring sentences are about different things, their embeddings will be far apart, and that is where to split.
It can help on messy text where topic shifts do not match paragraph breaks, such as a transcript where speakers jump between subjects. But the evidence is mixed, so test it against recursive splitting before paying for it. In Chroma's study, one popular semantic chunker scored lower recall than plain recursive splitting, while an embedding-based chunker Chroma designed itself scored a little higher. A 2024 study across three retrieval tasks concluded that the computational cost of semantic chunking is not justified by consistent gains (Qu et al. 2024, Is Semantic Chunking Worth the Computational Cost?).
The cost is real. You must embed every sentence in your collection before you can even start chunking, and the similarity threshold has to be tuned for each kind of document. For our handbook, semantic chunking is unnecessary: the headings already mark every topic boundary.
How does overlap actually help retrieval?
Overlap means each chunk repeats some text from the end of the one before it. If chunk 1 covers tokens 1 to 500 and the overlap is 100, chunk 2 starts around token 400.
It targets a specific problem: an answer that straddles a boundary. Without overlap, neither chunk contains the whole answer. With overlap, the boundary region appears in both, so a short fact near the boundary appears whole in at least one chunk.
Overlap also helps sentences near a boundary keep their context. A sentence that begins "This policy does not apply to contractors" means little without the sentence before it, which says which policy.
The tradeoff is storage and prompt space. Overlap increases the number of tokens you index; 20 percent overlap means roughly 20 percent more. It also means the retriever can return near-duplicate chunks that waste space in the prompt. And as the demo above showed, a splitter that works in whole pieces may repeat nothing at a paragraph break. Many teams start with 10 to 20 percent overlap, but treat that as a convention to test: in Chroma's study, overlap did not improve recall.
What is contextual retrieval and why does it change the chunking equation?
Contextual retrieval tackles a basic limitation of chunking: each chunk is embedded on its own, without knowing which document it came from. A chunk that says "The rate increases to 15% after the first year" is ambiguous. Fifteen percent of what? Contextual retrieval asks an LLM to write a short note that places each chunk within its document, and prepends that note to the chunk before embedding it and before indexing it for keyword search (Anthropic's contextual retrieval post).
The process works like this. For each chunk, you build a prompt with the whole document and the chunk, and ask the LLM for a brief description of where the chunk fits. You prepend that description to the chunk, then embed the combined text.
Anthropic reported that contextual embeddings alone cut the rate of failed retrievals in the top 20 chunks by 35 percent, from 5.7 to 3.7 percent. Adding contextual BM25, the same notes applied to keyword search, cut failures by 49 percent, and adding a re-ranker on top brought the reduction to 67 percent. These gains come without moving any chunk boundaries.
Our handbook's heading prefix is a cheap, rule-based cousin of this idea. Real documents need more: a chunk from the benefits section might say "Coverage begins on the first of the month following 30 days of employment", and only a note like "This describes when health insurance starts for new full-time employees" tells the embedding model what kind of coverage it means.
The cost is an extra LLM pass over every chunk at indexing time. For a handbook that changes a few times a year, that is small. For a collection that changes daily, it adds up, so weigh your update rate against the quality you need.
What does late chunking do differently?
Late chunking reverses the order of operations. Instead of chunking first and embedding each chunk alone, you run the whole document through a long-context embedding model first, then cut the resulting token-level vectors into chunks and average each group (Jina AI's late chunking post, Günther et al. 2024, Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models).
Because the model saw the entire document while producing each token's vector, every token carries some of the document's context, and so does the averaged chunk vector. A sentence like "As described above, this exception only applies in California" gets a vector that reflects what "above" referred to, even if the chunk boundary leaves it out.
The downside is cost and tooling. You need an embedding model whose input limit fits your longest documents, you pay to embed whole documents, and you need access to the token-level vectors, which not every embedding API offers.
Retrieve small, read big
There is one more fix for policies that get split, and it is simple. Search over small chunks, because small chunks match questions precisely, but when one matches, hand the model the larger section it came from. This is often called parent-document or small-to-big retrieval. In our handbook, a question that matches the accrual-rate sentence would pull in the whole vacation section, waiting period included. The cost is a longer prompt, so keep the parent sections modest in size.
How do you evaluate whether your chunking strategy is working?
You cannot evaluate chunking in isolation. You evaluate it through end-to-end retrieval and answer quality. The standard approach is a set of questions with known answers and known source passages, run through your system, measuring whether the right text was retrieved and whether the answers were correct.
The trap is that retrieval metrics like recall@k are sensitive to chunk boundaries. If you label "the right chunk" for each question under one chunking strategy, then switch strategies, the labels no longer line up, and recall can look worse even though the information is still retrievable. Either label at the level of passages or facts, so the labels survive a change of chunking, or accept that your metrics are relative to your chunking.
Chunk size also shapes what the numbers mean. Tiny chunks can score well on "was the right chunk retrieved?" while answers stay poor, because no single chunk holds enough context to answer. Large chunks almost always contain something relevant, so recall looks great while the needed fact sits buried among irrelevant text. That is why you compare strategies on both retrieval metrics and answer accuracy. If strategy A has higher recall but lower answer accuracy than strategy B, A's chunks are probably too small to carry whole rules.
For our handbook assistant, the evaluation should include questions that combine sections ("Do California employees get more sick leave than others?") and questions that turn on exceptions ("Are contractors eligible for the extra sick leave?"). Those are the questions that expose chunking failures. Part 7 builds this evaluation in code.
Quick Reference
| Property | Value |
|---|---|
| Default LangChain separators | ["\n\n", "\n", " ", ""] |
| Common starting chunk size | 256 to 512 tokens, a convention to test; NVIDIA found factoid questions did well at 256 to 512 |
| Common starting overlap | 10 to 20 percent, a convention to test; Chroma saw no recall gain from overlap |
| Measure chunk size with | The tokenizer of your embedding model |
| all-MiniLM-L6-v2 input limit | 256 word pieces, longer input is truncated |
| bge-small-en-v1.5 input limit | 512 tokens |
| OpenAI text-embedding-3-small and -large input limit | 8,192 tokens (as of October 2026) |
| Contextual retrieval, fewer failed retrievals | 35% (embeddings), 49% (with contextual BM25), 67% (with a re-ranker) |
Frequently Asked Questions
Q: Should I use token-based or character-based chunk sizes?
Token-based. Embedding models and LLMs count tokens, not characters, and the same number of characters can be very different numbers of tokens depending on the text. Use the tokenizer of the embedding model you will use, as chunking.py does, so a chunk never exceeds what the model reads. LangChain's splitters count characters unless you give them a token-based length function.
Q: How do I handle tables and lists during chunking?
Badly, if you treat them as ordinary text. Tables lose their structure when flattened into a stream, and a boundary can slice a row in half. Extract tables as structured data, convert each into a text form that keeps rows and columns together (such as a Markdown table), and treat each table as one chunk. Do not let the splitter break a table across chunks. For PDFs, use a parser that recognizes tables and headings rather than one that returns plain lines of text.
Q: Does chunk overlap affect embedding cost?
Yes. Overlap increases the total number of tokens you embed and store. A 20 percent overlap means roughly 20 percent more tokens than none. Measure whether it improves your retrieval before paying for it; in Chroma's study it did not.
Q: Can I use different chunking strategies for different document types in the same system?
Yes, and you often should. A handbook section, a code file and a chat transcript have different structures. Split the handbook by headings, the code by functions or classes, and the transcript by speaker turns or topic. Store the source type as metadata on each chunk; the retriever does not care how a chunk was made.
Q: How often should I re-chunk my collection?
Whenever your chunking strategy changes, you must re-chunk and re-embed everything. Old embeddings encode the old boundaries, and mixing old and new chunks in one index gives inconsistent retrieval. If you are iterating on chunking, budget for a full re-index each time.
Test yourself
Your handbook assistant uses recursive character splitting with 512-token chunks and 10 percent overlap. A user asks: "What is the vacation policy for employees in their first year?" The retriever returns three chunks. Chunk 1 holds the first half of the vacation section. Chunk 2 holds the second half, including "Employees in their first year accrue 1 day per month." Chunk 3 is from the sick leave section and is irrelevant. The LLM answers: "Employees in their first year accrue 1 day of vacation per month." But the full policy says first-year employees only start accruing after completing 90 days of employment, and that condition was in chunk 1. The LLM saw both chunks. Why did it still get the answer wrong, and what chunking change would most likely fix it?
Answer: Even with both chunks in the prompt, the model has to connect a condition in one chunk with a rate in another, and the rate is the sentence that matches the question best. It may give the explicit rate from chunk 2 and miss the condition from chunk 1, especially if other text sits between them. The root cause is that a boundary split one policy into two pieces that must be read together, and the overlap was too small to carry the condition across. The most reliable fix is to keep each policy whole: split on the handbook's headings, as chunking.py does, and only split further when a section exceeds the size limit. Another fix is small-to-big retrieval: search small chunks, but give the model the whole section a matching chunk came from. Either way, the waiting period and the accrual rate reach the model together.
Next: Embeddings and Vector Search, Demystified. It turns these chunks into embeddings, searches them, and shows when exact search is enough and when you need an approximate index.
If you want this kind of breakdown every week, how real RAG systems actually work under the hood, not just the tutorial version, subscribe to Internals Decoded at internalsdecoded.com.
Sources
- Chunking Strategies for LLM Applications (Pinecone)
- Splitting recursively (LangChain, read October 2026)
- Split markdown (LangChain, read October 2026)
- Evaluating Chunking Strategies for Retrieval (Chroma, July 2024)
- Finding the Best Chunking Strategy for Accurate AI Responses (NVIDIA, June 2025)
- Is Semantic Chunking Worth the Computational Cost? (Qu et al. 2024)
- Introducing Contextual Retrieval (Anthropic, September 2024)
- Late Chunking in Long-Context Embedding Models (Jina AI, August 2024)
- Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models (Günther et al. 2024)
- all-MiniLM-L6-v2 model card (Hugging Face)
- bge-small-en-v1.5 model card (Hugging Face)
- Vector embeddings guide (OpenAI, read October 2026)
- Qwen3-Embedding model card (Hugging Face)
