Agentic RAG: When the Retriever Starts Thinking
Query rewriting, multi-step retrieval, and agents that decide what to look up.
In the last episode we measured retrieval quality. Now we make the retriever care about those metrics. Agentic RAG replaces the one-shot “retrieve-then-read” pipeline with a loop where the language model calls retrieval tools, grades the results, rewrites the query if needed, and stops only when it has enough context. The key systems are a decision-making LLM (large language model), a suite of retrieval tools, and feedback nodes that catch and correct retrieval failures before they reach the answer.
Here is the surprising bit: a retriever that actively chooses what to fetch can be both more accurate and cheaper. A-RAG, for example, consistently beats static top-k pipelines on open-domain QA (quality assurance) benchmarks while retrieving fewer tokens. The agent learns to skip irrelevant context and focus on exactly the passages that connect the dots.
If you trained a human assistant to answer questions from your company handbook, you would not tell them “always hand me the three most similar pages and I will read from that.” You would expect them to decide when to search, what to type into the internal search bar, whether the results look promising, and when to rephrase the question. That is the mental model for agentic RAG.
A traditional RAG system acts like a junior clerk following a rigid script. The query is embedded, the top k chunks are retrieved, and the LLM generates an answer without ever questioning the quality of the context. When the query is ambiguous or the first batch of chunks misses critical information, there is nothing the system can do. Agentic RAG gives the LLM the tools and the authority to act like a senior researcher: it can plan a multi-step search, inspect intermediate results, and loop back if the first attempt fails.
What makes retrieval “agentic” in RAG?
In agentic RAG, the language model is given tools to retrieve information and the authority to decide when and how to use them. No engineer hard-codes the retrieval logic. The model itself chooses among keyword search, semantic search, chunk-level reads, and even web fallback, all while tracking what it has already seen. Research from IBM and practitioner frameworks like LangGraph describe this as adding memory, planning, and tool-calling capabilities to the retrieval step, transforming the retriever into a stateful actor that updates its plan based on observations IBMLangGraph.
The shift becomes concrete when you look at the company handbook example. A static RAG system would embed the question “What is the expense policy for international remote workers?” and retrieve the three chunks closest in vector space. Those chunks might cover the broad remote work policy but completely miss the paragraph about international exceptions hidden inside a different document. In an agentic setup, the LLM first parses the question and decides it needs information from at least two sources: the remote work policy and the international employment addendum. It can then issue two separate retrieval calls, perhaps using metadata filters to target the correct sections. If the retrieved text for international exceptions is too vague, the agent can rewrite that sub-query and search again.
The control loop looks like this inside a graph-based framework such as LangGraph. A router node uses the LLM to decide whether the query is answerable from the model’s own knowledge or requires retrieval. If retrieval is needed, the node emits a tool call to a retriever function. The retriever returns documents as a tool message. A grade node examines those documents and decides if they are relevant and sufficient. If not, a rewrite node reformulates the query and routes back to the retriever. Once the grade node approves the context, a generate node produces the final answer conditioned on the original question and the vetted documents LangGraph tutorial.
The loop terminates the moment the grader is satisfied. This design keeps the system from burning compute on unnecessary retrievals when the first search is already good enough. It also prevents the answer from being generated with bad context. Every cycle through the loop refines the agent’s understanding of what it still needs to find.
How does an agentic retriever plan and execute multi-step searches?
Planning in agentic RAG means decomposing a question into sub-queries and deciding which tools to call for each sub-goal. For the handbook assistant, a question like “Which managers can approve international travel budget over $5,000?” might require first retrieving the travel approval hierarchy, then the definition of an international trip, and finally any budget thresholds from the finance policy. A static system would try to answer from whatever chunks happen to be most similar to the entire sentence; an agent can break the problem apart and retrieve in order.
The A-RAG architecture makes this explicit. It gives the agent three retrieval tools: keyword search, semantic search, and chunk-read. The agent might start with a keyword search for “travel approval hierarchy” to get the right document, then use semantic search within that document to find passages about managerial limits, and finally call chunk-read to pull the exact numbers A-RAG. Each tool returns observations that the agent uses to decide the next step. In benchmark evaluations, models that select tools hierarchically achieve higher accuracy while retrieving fewer tokens, because they avoid pulling whole documents when a single paragraph would do A-RAG.
Multi-step planning can also be designed as a one-time scheme. The E-Agent framework trains a planner that lays out the entire sequence of tool calls upfront, then executes them without mid-flight replanning. On multimodal tasks this approach reduces redundant searches by 37% while improving accuracy by 13%, showing that even a static plan generated by the LLM can be highly efficient when it correctly anticipates dependency chains E-Agent. For the handbook assistant, a one-time plan might look like: (1) keyword search “travel budget,” (2) semantic search “international definition” within results of step 1, (3) chunk-read the finance policy section on dollar limits. The execution engine follows that plan, feeding the output of each step into the next.
Reflection and grading are what keep the plan on track. After each retrieval, a grader node checks whether the returned content actually addresses the sub-goal. If it does not, the agent can rewrite that sub-query or switch tools. In a company handbook assistant, if a semantic search for “international definition” returns the password reset page, the grader flags it as irrelevant, triggering a rewritten query like “what counts as an international trip for travel policy purposes.” This loop is the mechanism behind what surveys call “reflection patterns” agentic RAG survey. The agent is not blindly following a script; it evaluates its own results and adapts.
Why does agentic RAG avoid the failures of naive retrieval?
Naive retrieval systems suffer from three failure modes that agentic RAG directly counters: blind trust, static context windows, and single-hop limitations. Blind trust is the assumption that whatever the retriever returns is useful. The Corrective RAG paper documents cases where the generator faithfully produces an answer from irrelevant context, even when a human reader would immediately see the mismatch CRAG. Agentic RAG inserts a grading step between retrieval and generation. If the grade is low, the context is discarded and the query is rewritten. The answer is never built on bad evidence.
Static context windows hurt when the question requires more than k chunks. Suppose the handbook assistant answers a multi-part question about benefits eligibility. A static top-10 retrieval might pull chunks about health insurance but miss the separate page about retirement contributions. An agentic system can recognize that it needs more documents and issue a second retrieval specifically for retirement contributions, concatenating the two sets before generation. This dynamic expansion is what DataCamp calls “dynamic retrieval” DataCamp. The retriever stops being a fixed-size window and becomes a demand-driven mechanism that fetches only what the generator actually needs.
Single-hop limitations arise when the answer requires chaining facts from two or more documents. In the travel budget example, the approval hierarchy might be in the manager handbook, the international definition in the travel policy, and the dollar threshold in the finance guide. No single chunk contains the full answer. A naive retriever that searches only once with the original question will struggle to surface all three. Agentic RAG handles this by planning a multi-step retrieval, storing the intermediate facts in its state, and synthesizing them at the end. Research on multi-hop question answering confirms that multi-step retrieval strategies dramatically improve accuracy on such queries multi-hop retrieval. The agent becomes a bridge builder, not a single-hop router.
- Retrieves top k chunks
- Misses cross document facts
- Returns incomplete answer
- Decomposes into sub queries
- Retrieves from multiple sources
- Grader ensures completeness
How do you implement agentic RAG for a company handbook assistant?
Building this for the handbook assistant requires a stateful orchestration layer, a set of index-specific retrieval tools, and a grader that knows when context is insufficient. In LangGraph, you define a graph with nodes for routing, retrieval, grading, rewriting, and generation. The state carries messages and a list of retrieved documents. The retrieval node calls a function that wraps your vector store, keyword index, or file-level retriever and returns Document objects. The grader node might be a small LLM call that rates the relevance of each document on a 1-5 scale, and the rewrite node uses the LLM to produce a more focused query based on what was missing.
A concrete flow for the handbook might use LlamaIndex’s agentic retrieval composite retriever. You first create three sub-retrievers: a chunk-level semantic retriever, a file-via-metadata retriever for queries that mention specific document names, and a file-via-content retriever for broad searches. The composite retriever uses an LLM-based classifier to route the initial query to the appropriate sub-retriever. Inside each branch, you can still apply the grade-rewrite loop.
For example, if the user asks “Show me the parental leave policy from the HR handbook,” the router sends the query to the file-via-metadata retriever with a filter on “HR handbook.” The retriever returns the full file. A grader can then confirm that the file mentions parental leave. If the query were instead “What is our stance on working from a different country?” and no single file name is mentioned, the router might use file-via-content retrieval to find the best-matching full document, then chunk retrieval inside that document to pull the specific passage. The hybrid approach keeps the agent from drowning in irrelevant chunks.
The grading step is the most important safety feature. In production, you might use a dedicated RAG evaluation model like a cross-encoder to score every retrieved passage. If the average score falls below a threshold, the agent routes to a rewrite node that prompts the LLM to produce a clarified question. This process can be instrumented so that you log every retrieval attempt and its grade. Over time, you can feed failures back into prompt templates or even fine-tune a small router model that learns which retrieval strategies work for which kinds of questions DataCamp.
Quick Reference
| Property | Value |
|---|---|
| Core control loop | Route → Retrieve → Grade → Rewrite (if needed) → Retrieve again → Generate |
| Retrieval tools (A-RAG style) | Keyword search, semantic search, chunk-read |
| LlamaIndex agentic retrieval modes | Chunk, files_via_metadata, files_via_content, auto_routed |
| Key LangGraph nodes | generate_query_or_respond, retrieve, grade_documents, rewrite_question, generate_answer |
| Grading implementation | LLM-as-judge or cross-encoder model scoring each document |
| Feedback loop | Rewrites query based on grader output; can update router policies over time |
| Primary research demonstrating accuracy gain | A-RAG: higher accuracy with fewer tokens A-RAG; E-Agent: 13% accuracy gain, 37% fewer redundant searches E-Agent |
Frequently Asked Questions
Q: How does agentic RAG handle multi-turn conversations where the user follows up on a previous answer?
The agent maintains a message history in its state. When a follow-up arrives, the planner can reference earlier retrievals and avoid re-fetching the same documents. If the user asks “Can you give me the details on that exception you mentioned?”, the agent retrieves only the specific detail, not the whole prior context.
Q: Does an agentic retriever need a separate model or can the same LLM handle routing, grading, and generation?
Most implementations use a single LLM for all reasoning steps, with function-calling for tools. Grading is often done with the same LLM via a classification prompt. However, performance-sensitive systems sometimes use a smaller, fine-tuned model for routing and grading to reduce latency and cost.
Q: How do you prevent an agentic RAG system from getting stuck in a retrieval loop?
The control loop is designed with a maximum iteration count and a grader that can accept “no good context” as a valid state. If the grader repeatedly marks results as irrelevant, the agent falls back to a predefined answer like “I couldn’t find enough information in the handbook” rather than looping forever.
Q: Can agentic RAG work with hybrid search or does it replace the need for keyword and vector search together?
Agentic RAG augments hybrid search by letting the model choose which mode to use, but it does not replace the underlying retrieval machinery. You still need both dense and sparse indexes. The agent simply decides when to call one, the other, or both, depending on the query.
Q: Is agentic RAG only useful for large, heterogeneous document sets, or does it help with a single dense handbook too?
It helps with any corpus where questions vary in complexity. Even a single handbook can contain multi-hop questions and ambiguities. The agentic loop prevents the system from answering with the wrong section because the grader catches mismatches, irrespective of corpus size.
Test Yourself
A user asks your company handbook assistant: “What is the cap on home-office equipment reimbursement for employees in Germany?” Your agentic system retrieves the global equipment policy chunk, which mentions no country-specific cap, and the grader rates it as relevant. The answer confidently states “There is no cap,” but a German-specific addendum buried in a PDF actually sets a €500 limit. The user later reports the error. What went wrong in your agentic loop, and how would you fix it without abandoning the agentic pattern?
Answer: The grader rated the global policy as relevant because it addressed equipment reimbursement, but it lacked the signal that country-specific rules might override it. The fix is to add a post-grading verification step that checks for jurisdiction-specific documents whenever the query mentions a country. This could be a rule-based filter in the retrieval node: if a country name is detected, the agent must also call a tool that searches for region-specific addenda, perhaps using metadata filters. Alternatively, you can add a second grader that explicitly asks “Does the context mention any country-specific exceptions that are missing?” and, if not, triggers a rewrite that appends the country name. The loop structure remains intact. You are just making the grader more aware of the hierarchy of documents in your corpus.
If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com. Next week we start a new series.
Sources
- IBM: What is agentic RAG?
- LangGraph agentic RAG tutorial
- A-RAG: Agentic Retrieval-Augmented Generation
- E-Agent: Efficient Multimodal RAG with One-Time Planning
- CRAG: Corrective Retrieval-Augmented Generation
- Agentic RAG survey
- Multi-hop retrieval and multi-step RAG
- DataCamp: What is agentic RAG?