Context Window Management for Big Jobs
Chunking, map-reduce, and retrieval: how agents handle work that never fits in one context window.
In Part 7 we added hallucination detection layers so Mailmind could trust its own answers. But trust is useless if the agent never saw the information in the first place. This article is about what happens when the raw input is too big for one model call.
Mailmind has to process a year's worth of subscriptions, find a hidden flight change fee, and reschedule meetings across 50-message threads. The raw text does not fit in one context window. Even when it does, the model's attention fades in the middle. So you need strategy. The core strategies are chunking, map-reduce, retrieval, and hierarchical processing. Each splits the work, processes the pieces, and merges the results. The right choice depends on whether you need completeness, per-item precision, or long-range dependencies.
Why does model quality degrade before the context limit?
A model's context window is like a 500-page report you read in one sitting. By page 300, you are skimming. The same thing happens to transformers. Researchers call it the "lost in the middle" problem. The model pays strong attention to the beginning and the end of a long prompt. The middle gets fuzzy treatment.
For Mailmind, that means a flight change rule buried in message 25 of a 50-message thread might get weak attention. The numbers fit inside the context window. The attention did not. The agent never raises an error. It just produces an answer that ignores the middle content. This is a silent failure mode. Hard truncation is easier to detect because the model never sees the missing messages. But soft degradation from lost-in-the-middle is harder to spot. Both are problems you have to manage.
So you cannot just cram everything in. You need to split the work.
How does chunking let an agent process more than it can hold?
Chunking splits the raw input into smaller pieces before the model ever sees them. Each piece is processed independently. The results are merged later. The simplest version is fixed-size chunking. You cut the text into blocks of roughly 1000 to 2000 tokens with a 10 to 20 percent overlap. The overlap prevents a single fact from being split across two chunks where neither half makes sense.
Mailmind uses chunking on long email threads. Each email becomes its own chunk. That is semantic chunking. You split at natural boundaries instead of fixed token counts. An email about a flight change is a logical unit. A thread with 40 messages becomes 40 chunks. The overlap still matters. You might include the subject line and the previous message's first sentence in each chunk to keep the conversation context.
The sweet spot for chunk size is a tradeoff. Too small and you lose context. Too large and you hit the same middle-degradation problem inside the chunk. For Mailmind's email processing, one message per chunk works well. For a 100-page PDF attachment, you might use fixed-size chunks of 1500 tokens with a 15 percent overlap.
Chunking alone works for tasks that treat each piece independently. Finding all unread emails, tagging spam, extracting dates. But if a task needs the whole picture, a flat chunking approach falls short. That is where hierarchical processing comes in.
How does hierarchical processing handle large-scale tasks?
Hierarchical processing does a coarse pass first, then a deep dive. The coarse pass builds an outline or a category map. The deep dive only looks at the relevant sections. This keeps the total context use small while still getting full detail where it matters.
Mailmind uses hierarchical processing for its weekly "subscription triage" job. The inbox has 500 emails. The agent needs to find all subscription-related messages and decide which ones to cancel, which to renew, and which to negotiate. A flat chunking approach would process every email with the same depth. That is wasteful. The coarse pass scans every email with a lightweight model or a fast keyword filter. It marks each email as "subscription" or "not subscription." Then the deep dive processes only the subscription emails with full attention.
The coarse pass can use large chunks of 10 emails at a time. The deep dive uses one email per chunk. The overall cost is a fraction of processing everything at full depth. The tradeoff is that the coarse pass might miss something. If a subscription email uses odd phrasing, it might get filtered out. That is a recall versus precision tradeoff. You can tune the coarse pass to be more aggressive, but then you spend more on the deep dive.
Hierarchical processing works best when you have a clear two-tier structure. You need to find the needles, then examine them. If the task is more of a whole-corpus summary, map-reduce is a better fit.
How does map-reduce summarize an entire inbox?
Map-reduce borrows its name from the distributed computing pattern. You split the work into independent map tasks. Each map task processes its chunk and produces a partial result. Then a reduce step combines those partial results into a final answer. The map step never needs to see the whole corpus. The reduce step only sees the partial results, not the raw text.
Mailmind uses map-reduce when a user asks, "Summarize all my subscriptions this year." The inbox has 200 subscription emails. The agent splits them into 20 chunks of 10 emails each. Each map agent extracts the service name, the monthly cost, and the next renewal date from its chunk. The reduce agent takes all 20 partial lists and produces a categorized summary: "Netflix: $15.99/month, renews Aug 3. Spotify: $11.99/month, renews Sep 1." The map agents never see the full list. The reduce agent never sees the raw emails. The total token consumption is far lower than processing all 200 emails in one massive prompt.
The risk with map-reduce is that the partial summaries lose nuance. If a map agent extracts only the dollar amount but not the fact that it is a promotional rate that expires in 3 months, the reduce agent never sees that detail. The fix is to design the map output schema carefully. Include a "notes" field for edge cases. Another risk is that the reduce agent hallucinates connections between the partial results. That is a hallucination detection problem, but it is amplified when the reduce agent has no raw text to verify against.
Map-reduce scales to any size. You can process a million emails with enough map agents. The reduce step is the bottleneck, but it only sees structured data, not raw text. For tasks that are more about retrieving a specific fact, retrieval is more efficient.
What role does retrieval play in context management?
Retrieval turns the problem inside out. Instead of processing everything, you embed the chunks and store them in a vector database. When a query comes in, you retrieve only the 3 to 5 most relevant chunks. Those chunks go into the context window. The rest is ignored. This is the most token-efficient strategy.
Mailmind uses retrieval for targeted questions like "What is the change fee for my flight to Boston?" The inbox has thousands of emails. The retriever finds the 3 chunks that mention "change fee," "Boston," and "flight." The agent only sees those 3 chunks. The answer costs a few hundred tokens instead of tens of thousands. The retrieval step is fast. The embedding model runs once per chunk, and the vector search is near-instant.
The catch is that retrieval works only for questions that can be answered from a few chunks. If the user asks a comparative question like "Which subscription has the best value?" the retriever might not pull the right chunks. You need a map-reduce or a hierarchical approach for that. Retrieval also depends on the quality of the embeddings. If the chunks are embedded poorly, the retriever misses relevant information. Semantic chunking helps here too. Emails that talk about the same topic but use different words still get retrieved if the embeddings capture the meaning.
Retrieval and map-reduce are not mutually exclusive. You can use retrieval to find the relevant chunks, then map-reduce over those chunks. That is a hybrid that works for many real-world tasks. For long-range dependencies inside a single document, sliding windows and progressive summarization are the tools.
How do sliding windows and progressive summarization preserve long-range dependencies?
Some tasks need to track a thread that spans many messages. A project discussion might have decisions in message 1, objections in message 15, and a resolution in message 30. A flat chunker would split this into chunks that lose the cross-references. A sliding window processes the thread with overlapping windows. Each window covers a few messages and shifts by half a window. That way, every message appears in at least two windows, and the agent can see the connections.
Mailmind's meeting scheduler uses a sliding window for long threads about date and time proposals. A window of 4 messages slides by 2 messages. The agent in each window extracts the relevant proposals and objections. A final synthesis step combines the findings. The overlap ensures that a proposal in message 3 and a counterproposal in message 5 are seen together in the window that covers messages 2 to 6.
Progressive summarization takes a different approach. It summarizes sections of the thread, then summarizes the summaries. The final answer is built from a hierarchy of summaries. This is useful when the thread is too long even for sliding windows. Mailmind uses it for threads that span hundreds of messages. The first pass summarizes each message into a sentence. The second pass groups 10 messages and summarizes the group. The third pass groups the groups. The final summary is a few paragraphs. The agent can then answer "What was decided about the Q3 budget?" by looking at the top-level summary.
The tradeoff is that progressive summarization loses detail. A specific number buried in a message might get smoothed over in the summary. That is why Mailmind always keeps the original chunks for retrieval. The summary is for coarse understanding. The raw chunks are for fact-checking.
Both strategies have to handle the same boundary problem that chunking does. When a fact is split across a window boundary or a summary boundary, the agent can miss it. That is the boundary bug.
What are boundary bugs and how do you fix them?
A boundary bug happens when a logical unit of information is split across two processing chunks. The agent in each chunk sees only a fragment. Neither fragment is enough to act on. The result is a silent failure. The agent never errors. It just never produces the right answer.
Mailmind hit this with a booking confirmation thread. The flight details were in message 3. The hotel details were in message 7. The change fee rules were in an attachment summary in message 14. The chunker split the thread into two chunks: messages 1 to 8 and 9 to 20. The map agent for the first chunk saw the flight and hotel but no change fee. It flagged "incomplete trip information." The map agent for the second chunk saw the change fee but no flight details. It also flagged "incomplete." The reduce agent got two incomplete flags and produced a generic "I could not find the full trip details" response. The user never got the itinerary.
The fixes are layered. The first layer is bigger overlap. If the overlap had been 4 messages instead of 2, the two chunks would have shared more context. The second layer is semantic chunking. Instead of splitting by message count, split by logical sections. The attachment summary in message 14 is a section. The chunker should treat it as a unit and attach it to the thread that it references. The third layer is a boundary detection pass. Before processing, scan the thread for cross-references. The phrase "see attached fare rules" in message 3 tells the system to link message 3 and the attachment. The chunker can then process them together.
The fourth layer is coverage tracking. After processing, Mailmind checks that every message in the thread was assigned to at least one chunk and that every logical entity (trip, subscription, meeting) was fully processed. If any entity is flagged as incomplete, the system reruns the processing with a different chunking strategy. Silence is not success. You have to verify that the work was actually done.
- Fare rules in msg 35 not shared
- Attachment context in msg 38 missed
- Cross-reference undetected
- Messages 35 to 38 within shared context
- Fare rules extracted from attachment
- Cross-reference resolved
These fixes make the system robust. They turn silent failures into explicit retries. The cost is extra computation, but for critical tasks like trip planning, the cost is worth it.
Quick Reference
| Strategy | Best for | Token range | Overlap needed? |
|---|---|---|---|
| Fixed-size chunking | General splitting | 1000 to 2000 tokens per chunk | 10 to 20 percent |
| Semantic chunking | Natural boundaries | Variable | No |
| Map-reduce | Summarization over a corpus | Any size | No |
| Retrieval (RAG) | Targeted queries | Retrieve top 3 to 5 chunks | No |
| Hierarchical processing | Large-scale analysis | Coarse pass: large chunks; fine pass: focused | No |
| Sliding window | Long-range dependencies | Window size 2000 to 4000, shift 50 percent | Yes |
| Progressive summarization | Deep dive on long docs | Summaries of sections, then combine | No |
Frequently Asked Questions
Q: How do I choose between chunking and retrieval for a given task? Use retrieval when the task is a specific question that can be answered from a few chunks. Use chunking when the task requires processing every piece of input independently, like extracting all dates or tagging all messages. The two can be combined: retrieve the relevant chunks, then process them.
Q: What is the biggest pitfall with map-reduce? The reduce step can lose nuance if the map outputs are too high-level. If a map agent extracts only the dollar amount but not the fact that it is a promotional rate, the reduce agent never sees that detail. Always include a notes field in the map output schema for edge cases.
Q: Can I just use a model with a huge context window and skip all this? Even with a 1 million token context window, the lost-in-the-middle effect still degrades the middle content. The cost and latency of filling a huge context are also high. For most tasks, a well-designed chunking or retrieval strategy is faster, cheaper, and more reliable.
Q: How do I detect boundary bugs in production? Implement coverage tracking. For each logical unit, like a thread or a document, verify that every piece has been processed. Log when an agent returns an "incomplete" flag. If the coverage check fails, escalate to a human or rerun with a different chunking strategy.
Q: When should I use hierarchical processing versus progressive summarization? Use hierarchical processing when you need to decide what to deep-dive based on a coarse pass. For example, finding all subscription emails and then analyzing them. Use progressive summarization when you need to reduce a long document into a dense summary without losing structure. Both are forms of multi-pass processing but serve different goals.
Test yourself
Mailmind receives a booking confirmation thread. The flight details are in message 3. The hotel details are in message 7. The change fee rules are in an attachment summary in message 14. Your chunker splits the thread into two chunks: messages 1 through 8 and 9 through 20. Each chunk's map agent flags "incomplete" because it sees only partial trip information. The reduce agent produces a generic "I could not find the full trip details" response. How would you redesign the pipeline?
Answer: Treat the entire thread as one logical document. Use a boundary detection pass to identify cross-references. The phrase "see attached fare rules" in message 3 links message 3 and the attachment. Process those messages together. Instead of fixed chunking, use semantic chunking that respects section boundaries. Let each agent write its findings to a shared store keyed by the thread ID. The hotel agent writes the hotel details. The flight agent writes the flight details. The attachment agent writes the change fee rules. A final reduce agent reads all entries from the store and synthesizes the complete itinerary. Coverage tracking verifies that messages 3, 7, and 14 were all processed. If any are missing, the system reruns with a larger overlap. This turns a silent failure into a verified complete answer.
If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.
Next episode: when latency matters more than correctness. Mailmind's real-time meeting scheduler cannot afford a 30-second pause. We will look at speculative execution, streaming, and the tradeoffs that keep an agent feeling instant.