Chunking: The Decision Everything Downstream Inherits

Chunking is the step where raw documents become the units your retriever searches and your LLM (large language model) reads. Every chunk boundary is a decision about what information stays together and what gets split apart. Get the sizes wrong, and you either flood the model with noise or starve it of context. Get the overlap wrong, and facts that span boundaries become invisible. Get the segmentation strategy wrong, and your carefully tuned retrieval pipeline is retrieving fragments that never had a chance of answering the question. Nothing downstream can fix a bad chunking decision because nothing downstream can see across the boundaries you drew at the start.

Here is the part that surprises engineers who have not debugged a RAG (retrieval-augmented generation) system in production: the chunking strategy you pick also determines what your offline evaluation metrics actually measure. If you evaluate retrieval accuracy by checking whether the "correct" chunk appears in the top-k results, you are really evaluating whether your chunk boundaries happened to isolate the answer in a single retrievable unit. A system with tiny chunks can score perfectly on retrieval benchmarks while producing terrible answers because no single chunk contains enough reasoning context. A system with large chunks can score terribly on retrieval benchmarks while producing great answers because the retriever always finds something relevant. Your metrics inherit your chunking decisions. So does everything else.

How does chunking fit into the RAG pipeline?

Chunking runs during indexing, after document parsing and before embedding. You take a parsed document, segment it into pieces, and each piece becomes the unit you embed, store, and later retrieve. The retriever never sees whole documents. It sees chunks. The LLM never sees whole documents. It sees whatever chunks the retriever pulled, stitched into a prompt.

This means chunking defines the granularity of your entire knowledge index. If a critical fact lives in your corpus but never appears in any single chunk without being diluted by irrelevant surrounding text, retrieval becomes a lottery. The embedding model might place that chunk near the query vector, or it might not. The similarity score reflects the average meaning of everything in the chunk, not the presence of one buried fact. source

Formally, a document is a sequence of tokens. A chunking strategy maps that sequence to a set of contiguous segments, each bounded by a maximum size. Each segment gets embedded independently. At query time, the user's question is embedded, and the system finds the chunks whose vectors are closest to the query vector. The top-k chunks, plus their text and metadata, become the context the LLM reads.

The pipeline does not know about relationships between chunks unless you explicitly encode them. If the answer to a question requires combining facts from chunks 3, 7, and 12, the retriever must surface all three. If it only surfaces two, the LLM works with incomplete evidence. If it surfaces twenty, the LLM drowns in noise and token costs balloon. Chunking controls the shape of this tradeoff before any retrieval logic runs.

Why can't we just use whole documents?

Two hard constraints make whole-document retrieval impossible for most real corpora. First, embedding models have fixed input limits. Older models cap out around 512 or 1024 tokens. Newer models stretch to 8,192 or more. But a single PDF manual or legal filing can run to hundreds of thousands of tokens. You cannot embed it as one unit. source

Second, even if you could embed a whole document, the resulting vector would be semantically meaningless. A 200-page document covers hundreds of topics. Its embedding would be a blurry average of all of them. A query about a specific configuration parameter would produce a similarity score that reflects the document's general subject matter, not the presence of that parameter. The retriever would rank documents by topical similarity, not by whether they contain the answer. That is useless for question answering.

There is a third, practical constraint: cost. Embedding and LLM API (application programming interface) calls are priced per token. If you stuff 50,000 tokens of context into every query when only 500 are relevant, you burn money on every request. Chunking lets you retrieve only the relevant slices.

So chunking is not optional. It is the mechanism that makes retrieval possible at all. The question is not whether to chunk. It is how.

What happens when chunk size is too small?

Small chunks give you precision. A 128-token chunk might contain exactly one definition, one instruction, or one data point. When the query matches that chunk, the similarity score is high and the retrieved text is focused. The LLM gets exactly what it needs and nothing else.

But small chunks also amputate context. Imagine your company handbook states: "Employees based in California receive an additional 24 hours of sick leave per year. This policy does not apply to contractors." If your chunk boundary falls between those two sentences, the retriever might surface only the first sentence. The LLM confidently tells a contractor they get extra sick leave. The system hallucinated because the chunking strategy hid the disqualifier. source

Small chunks also force you to retrieve more of them to cover the same amount of content. If you need five small chunks to capture what one well-sized chunk would have contained, you increase the odds that irrelevant chunks sneak into the top-k. You also burn more tokens assembling the prompt. And you increase the chance that the LLM must perform cross-chunk reasoning, which it is mediocre at, to connect facts that should have stayed together.

The NVIDIA benchmark study found that very small chunk sizes degraded performance across multiple datasets because chunks lacked sufficient context for both retrieval and generation. The embedding model could not distinguish between a chunk that contained a complete answer and a chunk that contained a sentence fragment.

What happens when chunk size is too large?

Large chunks preserve context. A 2,048-token chunk might contain an entire policy section, including definitions, conditions, and exceptions. When the retriever finds it, the LLM has everything it needs to reason correctly.

The problem is that large chunks dilute the embedding. A chunk that covers five distinct topics will have an embedding that represents the average of all five. A query about one specific topic might rank that chunk lower than a smaller, more focused chunk from a less relevant document. The signal gets washed out by the noise of everything else in the chunk.

Large chunks also waste tokens. If your retriever pulls three 2,000-token chunks to answer a question that only needs 300 tokens of context, you are paying to process 5,700 tokens of irrelevant text. At scale, across thousands of queries, that is real money. And if the irrelevant text is distracting enough, it can degrade answer quality by pulling the LLM's attention away from what matters.

There is a subtler failure mode. Large chunks make offline evaluation misleading. If your evaluation metric checks whether the "correct" chunk appears in the top-k, large chunks will almost always contain something relevant. Your recall numbers look great. But the LLM still produces bad answers because the relevant fact is buried in a sea of irrelevance. Your metrics say the system works. Your users say otherwise.

How does recursive character splitting actually work?

Recursive character splitting is the default strategy in LangChain and the starting point for most production RAG systems. It tries to keep paragraphs and sentences intact, only splitting at lower-level boundaries when a unit is too large to fit the target size. source

The algorithm takes a list of separators, ordered from highest-level to lowest-level. The default list is: double newlines (paragraph breaks), single newlines (line breaks), spaces (word boundaries), and finally the empty string (character boundaries). Given a text and a target chunk size, it first tries to split on double newlines. If any resulting segment is still larger than the target size, it recursively applies the next separator in the list to that segment. Only when it runs out of separators does it fall back to slicing at fixed character intervals. source

After this first pass, the algorithm has a list of pieces that are all under the target size. It then merges adjacent pieces greedily: keep adding pieces to the current chunk until adding the next piece would exceed the target size, then emit the chunk and start a new one. Overlap is handled by including a configurable number of characters from the end of the previous chunk at the start of the next one. source

Here is what this looks like in practice, using LangChain with a token-aware length function:

# LangChain v0.3.x
from langchain_text_splitters import RecursiveCharacterTextSplitter
import tiktoken

tokenizer = tiktoken.get_encoding("cl100k_base")

def tiktoken_len(text):
    return len(tokenizer.encode(text))

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=64,
    length_function=tiktoken_len,
    separators=["\n\n", "\n", " ", ""]
)

chunks = splitter.split_text(handbook_text)

The result is a set of chunks that respect paragraph and sentence boundaries whenever possible. A 512-token chunk will typically contain several complete paragraphs. The 64-token overlap means that if a sentence gets split because it falls at a chunk boundary, its first part appears at the end of one chunk and its second part appears at the beginning of the next. This gives the retriever two chances to find it.

The Chroma evaluation found that recursive character splitting with chunk sizes in the 400-512 token range and 10-20% overlap achieved recall in the 85-90% range across varied datasets. It is not the best strategy for every corpus, but it is the best default.

When should you use structure-aware chunking instead?

Structure-aware chunking uses the document's own organization to define boundaries. If your documents are Markdown with clear heading hierarchies, split at heading boundaries. If they are PDFs with logical page breaks, split at page boundaries. If they are HTML with semantic tags, split at section or article boundaries.

This works because document authors already grouped related content together. A subsection in a manual probably covers one coherent topic. A page in a report probably contains one complete idea plus supporting detail. By respecting these boundaries, you get chunks that are semantically coherent without needing an embedding model to detect topic shifts. source

The NVIDIA benchmark found that page-level chunking achieved the highest accuracy for paginated documents. This makes intuitive sense: a page is a unit the author designed to be read together. Splitting mid-page breaks that design. Keeping pages intact preserves it.

For our company handbook assistant, structure-aware chunking is particularly valuable. Handbooks are organized into sections with clear headings: "Vacation Policy," "Sick Leave," "Remote Work Guidelines." Each section is a natural chunk. Splitting mid-section risks separating a policy statement from its exceptions or eligibility criteria. Keeping sections intact means each chunk is a self-contained policy unit.

LangChain provides structure-aware splitters for Markdown, HTML, and code. The Markdown splitter, for example, splits on heading boundaries first, then applies recursive character splitting within sections that exceed the target size:

# LangChain v0.3.x
from langchain_text_splitters import MarkdownHeaderTextSplitter

headers_to_split_on = [
    ("#", "h1"),
    ("##", "h2"),
    ("###", "h3"),
]

splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=headers_to_split_on
)

chunks = splitter.split_text(markdown_handbook)

Each chunk inherits metadata about its heading hierarchy. You can prepend that metadata to the chunk text before embedding, so the vector representation includes structural context. A chunk from the "Sick Leave" section will be closer in embedding space to queries about sick leave, even if the chunk text itself does not repeat the phrase "sick leave."

What is semantic chunking and when is it worth the cost?

Semantic chunking uses embedding similarity to decide where topic boundaries fall. You split the document into sentences, embed each sentence, then compute the cosine similarity between consecutive sentence embeddings. When similarity drops below a threshold, you have found a topic boundary. You group sentences between boundaries into chunks, subject to size constraints. source

The intuition is clean: if two consecutive sentences are about different things, their embeddings will be far apart. That is where you should split. If they are about the same thing, their embeddings will be close. That is where you should stay together.

This approach outperforms fixed-size and recursive splitting on heterogeneous corpora where topic boundaries do not align with paragraph breaks. Think of a document that discusses multiple products in a single long paragraph, or a transcript where speakers jump between topics without clear structural markers. Semantic chunking can detect the shifts that structural splitters miss.

The cost is computational. You must embed every sentence in your corpus before you can even start chunking. For a large corpus, that is a significant indexing-time expense. You also need to tune the similarity threshold, which is dataset-specific. Too high, and you get tiny, fragmented chunks. Too low, and you get the same large, diluted chunks that fixed-size splitting produces.

For our handbook assistant, semantic chunking is probably overkill. Handbooks are well-structured documents with clear headings. Structure-aware splitting will capture most topic boundaries. Semantic chunking earns its keep on messy, unstructured text where no reliable structural signals exist.

How does overlap actually help retrieval?

Overlap means each chunk shares some tokens with its neighbors. If chunk 1 covers tokens 1-500, chunk 2 might cover tokens 400-900. The overlapping region, tokens 400-500, appears in both chunks.

This solves a specific problem: what happens when the answer to a query spans a chunk boundary? Without overlap, the answer is split across two chunks, and neither chunk contains the complete answer. The retriever might surface one chunk but not the other. The LLM sees half the answer and either guesses wrong or asks for clarification. With overlap, the boundary region appears in both chunks. If the answer falls in the overlap zone, either chunk can satisfy the query on its own. source

Overlap also helps with embedding quality at chunk boundaries. The first and last few sentences of a chunk often depend on context from neighboring chunks for full meaning. A sentence that begins "This policy, however, does not apply.." is meaningless without the preceding sentence that states the policy. Overlap ensures that boundary sentences appear with their necessary context in at least one chunk.

The tradeoff is storage and retrieval efficiency. Overlap increases the total number of tokens indexed. A 20% overlap on 512-token chunks means roughly 20% more tokens in your vector database. It also means the retriever might return chunks with substantial overlap, wasting tokens in the LLM prompt. Most production systems use 10-20% overlap as a reasonable balance.

What is contextual retrieval and why does it change the chunking equation?

Contextual retrieval addresses a fundamental limitation of chunking: each chunk is embedded in isolation, without awareness of the document it came from. A chunk that says "The rate increases to 15% after the first year" is ambiguous. Fifteen percent of what? Contextual retrieval prepends a short, LLM-generated description to each chunk before embedding. The description situates the chunk within the full document. source

The process works like this. For each chunk, you construct a prompt that includes the full document and the specific chunk. You ask an LLM to generate a concise description of what the chunk contains and how it relates to the document. You prepend that description to the chunk text, then embed the combined text. The resulting vector encodes both the local content and its document-level context. source

Anthropic reported that contextual embeddings alone reduced top-20 retrieval failure rates by 35%. Combining contextual embeddings with contextual BM25 (applying the same augmentation to sparse retrieval) reduced failures by 49%. Adding a reranker on top brought the total reduction to 67%. These are large improvements, and they come without changing chunk boundaries at all. source

For our handbook assistant, contextual retrieval is compelling. A chunk from the benefits section might say "Coverage begins on the first of the month following 30 days of employment." Without context, the embedding model does not know this is about health insurance versus life insurance versus something else. A contextual description like "This chunk describes when health insurance coverage begins for new full-time employees" makes the chunk far more retrievable for relevant queries.

The cost is an extra LLM pass over every chunk at indexing time. For a static handbook that changes quarterly, this is negligible. For a corpus that updates daily, it might be prohibitive. The tradeoff depends on your update frequency and retrieval quality requirements.

What does late chunking do differently?

Late chunking reverses the order of operations. Instead of chunking first and embedding each chunk independently, you embed the entire document first, then derive chunk embeddings from the token-level representations the model produced. source

The mechanism: you pass the full document through an embedding model with a large context window. The model produces a sequence of token embeddings, one per token. You then define chunk boundaries at the token level and pool the token embeddings within each boundary to produce a chunk vector. Because the model saw the entire document when producing token embeddings, each token's representation is informed by global context. The pooled chunk vector inherits that global awareness. source

This is powerful for documents where local passages are ambiguous without global context. A sentence like "As described above, this exception only applies in California" is meaningless if the chunk does not include "above." Late chunking ensures the token embeddings for that sentence already encode the referenced content, even if the chunk boundary excludes it.

The downside is cost and model requirements. You need an embedding model with a context window large enough for your longest documents. You pay to embed the full document, not just the chunks. And you need infrastructure that supports token-level embedding access, which not all embedding APIs provide. Late chunking is an advanced technique for systems where global context is critical and the budget supports it.

How do you evaluate whether your chunking strategy is working?

You cannot evaluate chunking in isolation. You must evaluate it through the lens of end-to-end retrieval and generation quality. The standard approach is to build an evaluation dataset of question-answer pairs with known ground-truth document sources, then measure whether your system retrieves the right chunks and generates correct answers.

The trap is that retrieval metrics like recall@k are sensitive to chunk boundaries. If your ground-truth answer is split across three chunks in your strategy but was contained in one chunk in the strategy used to build the evaluation dataset, your recall numbers will look worse even if the information is technically retrievable. You must either build evaluation datasets that are chunking-strategy-agnostic (by annotating at the passage or fact level) or accept that your metrics are relative to your chunking choices.

A practical approach: run multiple chunking strategies on the same evaluation dataset and compare both retrieval metrics and end-to-end answer accuracy. If strategy A has higher recall but lower answer accuracy than strategy B, your chunks are probably too small. The retriever finds the right pieces, but the LLM cannot assemble them into correct answers. If strategy B has lower recall but higher answer accuracy, your chunks are probably larger and more self-contained, which helps generation even if it hurts retrieval scores.

For our handbook assistant, the evaluation should include questions that require cross-section reasoning ("Do California employees get more sick leave than Texas employees?") and questions that depend on exceptions and qualifiers ("Are contractors eligible for health insurance?"). These are the questions that expose chunking failures.

Quick Reference

Frequently Asked Questions

Q: Should I use token-based or character-based chunk sizes?

Token-based. Embedding models and LLMs count tokens, not characters. A 512-character chunk might be 200 tokens or 400 tokens depending on the text. Token-based sizing gives you predictable context window utilization and predictable costs. Use your embedding model's tokenizer to measure chunk sizes.

Q: How do I handle tables and lists during chunking?

Badly, if you treat them as regular text. Tables lose all structure when flattened into a text stream, and chunk boundaries can slice rows in half. The better approach is to extract tables as structured data, convert them to a text representation that preserves row-column relationships (like Markdown tables), and treat each table as a minimum chunk unit. Do not let the splitter break a table across chunks.

Q: Does chunk overlap affect embedding cost?

Yes. Overlap increases the total number of tokens you embed and store. A 20% overlap on a 512-token chunk size means you embed roughly 20% more tokens than a no-overlap strategy. This is usually worth the retrieval quality improvement, but measure it against your budget.

Q: Can I use different chunking strategies for different document types in the same system?

Yes, and you should. A handbook section, a code file, and a chat transcript have different structures. Apply structure-aware splitting to the handbook, language-aware splitting to the code, and semantic chunking to the transcript. Your vector database stores chunks with metadata about their source type. Your retriever does not care how the chunks were made.

Q: How often should I re-chunk my corpus?

Whenever your chunking strategy changes, you must re-chunk and re-embed the entire corpus. Old embeddings encode the old boundaries. Mixing old and new embeddings in the same index produces inconsistent retrieval behavior. If you are iterating on chunking, budget for full re-indexing each time.

Test yourself

Your company handbook assistant uses recursive character splitting with 512-token chunks and 10% overlap. A user asks: "What is the vacation policy for employees in their first year?" The retriever returns the top 3 chunks. Chunk 1 contains the first half of the vacation policy section. Chunk 2 contains the second half, including the sentence "Employees in their first year accrue 1 day per month." Chunk 3 is from the sick leave section and is irrelevant. The LLM answers: "Employees in their first year accrue 1 day of vacation per month." But the full policy states that first-year employees accrue 1 day per month after completing 90 days of employment. The 90-day waiting period was in chunk 1. The accrual rate was in chunk 2. The LLM saw both chunks. Why did it still get the answer wrong, and what chunking change would most likely fix it?

Answer: The LLM saw both chunks but had to reason across them to combine the waiting period with the accrual rate. This is cross-chunk reasoning, and LLMs are unreliable at it. The model latched onto the explicit accrual rate in chunk 2 and either ignored or failed to integrate the constraint from chunk 1. The root cause is that the chunk boundary split a single policy into two pieces that must be read together. The fix is to increase chunk size so the entire vacation policy section fits in one chunk, or to switch to structure-aware chunking that respects section boundaries. A 512-token chunk is too small for this handbook's policy sections. Moving to 1,024-token chunks or using the Markdown header splitter to keep each policy section intact would keep the waiting period and accrual rate together, eliminating the cross-chunk reasoning requirement.

What comes next

Chunking decides what your retriever can find. But retrieval also depends on how you represent those chunks as vectors and how you search them. In the next episode, we will look at embedding models: how they turn text into vectors, why the choice of model changes what "similarity" means, and what happens when your embedding model does not understand your domain.

If you want this kind of breakdown every week, how real RAG systems actually work under the hood, not just the tutorial version, subscribe to Internals Decoded at internalsdecoded.com.

Sources

• Pinecone: Chunking Strategies for RAG
• LangChain: Recursive Text Splitter
• LangChain API: RecursiveCharacterTextSplitter
• Anthropic: Contextual Retrieval
• Jina AI: Late Chunking