Agentic RAG: When the Retriever Starts Thinking
Query rewriting, multi-step retrieval, and agents that decide what to look up.
In the last part we measured retrieval quality. Now we let the system act on what it finds. Agentic RAG replaces the one-shot "retrieve, then read" pipeline with a loop: the language model calls retrieval tools, judges what came back, rewrites the query if needed, and answers only when it has enough. The pieces are a decision-making LLM (large language model), a set of retrieval tools, and checks that catch weak results before they reach the answer.
The payoff is on hard questions, the ones that need several facts from different places or start out vague. The price is more model calls per question, and so more cost and waiting time. This part explains how the loop works, what the research shows it gains, what it costs, and when a plain pipeline is the better choice. It ends by putting all eight parts of the series together.
If you trained a person to answer questions from your company handbook, you would not tell them "always hand me the three most similar pages". You would expect them to decide what to search for, judge whether the results look right, and try a different search when they do not. That is the mental model for agentic RAG.
What makes retrieval "agentic" in RAG?
In agentic RAG, the model is given retrieval tools and the authority to decide when and how to use them. The engineer defines the tools and the loop; the model decides which tool to call and with what query. The tools might be a keyword search, a vector search, a "read this whole section" tool or a filtered search over one document type, and the model keeps track of what it has already seen. IBM's explainer describes agentic RAG simply as the use of AI agents to carry out retrieval-augmented generation (IBM's explainer on agentic RAG). If agents are new to you, The ReAct Loop: How Most AI Agents Actually Work explains the loop underneath.
Take the handbook question "What is the expense policy for international remote workers?" A static pipeline embeds the whole question and retrieves the closest chunks. They might cover remote work in general and miss the international exception in another section. An agent can split the question, search separately for the remote work policy and for the rules on working from another country, and if a search comes back vague, rephrase it and try again.
LangGraph's agentic RAG tutorial builds this as a small graph (LangGraph's agentic RAG tutorial). The first node, generate_query_or_respond, lets the model either answer directly or call the retriever tool. If it calls the tool, a retrieve node runs the search. A grading function, grade_documents, then asks the model whether the results are relevant to the question: if they are, the graph moves to generate_answer; if not, to rewrite_question, which rephrases the question and sends it back to the first node to search again.
For a grounded assistant like ours, restrict the "answer directly" branch to small talk such as "hi" or "thanks". Every policy question must retrieve. If the model is allowed to answer policy questions from its own knowledge, it will invent policy, and everything Part 6 built to keep answers honest is bypassed.
The loop ends when the grader is satisfied, so an easy question costs one search and a hard one gets more chances. Grading also keeps the answer from being built on results the grader judged irrelevant.
How does an agentic retriever plan and execute multi-step searches?
Planning means breaking a question into sub-questions and deciding which tool to call for each. "Which managers can approve an international travel budget over $5,000?" might need the approval hierarchy, the definition of an international trip, and the finance policy's budget thresholds. A static system retrieves whatever is closest to the whole sentence; an agent can fetch each piece in turn.
A-RAG makes this explicit. It gives the agent three tools at different levels of detail: keyword_search, semantic_search and chunk_read, and lets the model choose among them (Du et al. 2026, A-RAG: Scaling Agentic Retrieval-Augmented Generation via Hierarchical Retrieval Interfaces). The agent might use keyword search to find the right document, semantic search to find the relevant passages in it, and chunk read to pull the exact numbers. Each result shapes the next step.
How much does it help? On multi-hop question-answering benchmarks, which need facts from several documents, A-RAG with GPT-5-mini beat the other methods the authors compared. With a smaller model, GPT-4o-mini, it was not always ahead: HippoRAG2 scored higher on HotpotQA and on 2WikiMultiHopQA. And while the paper reports comparable or fewer retrieved tokens overall, its own table shows A-RAG reading about half the tokens of plain RAG on two datasets and more on three others.
Planning can also happen once, up front. The Efficient Agent framework, E-Agent, trains a planner that lays out the sequence of searches before running them. On multimodal RAG tasks with time-sensitive questions, such as news, it reported a 13 percent accuracy gain over the best earlier methods while cutting redundant searches by 37 percent (Wang et al. 2025, Efficient Agent: Optimizing Planning Capability for Multimodal Retrieval Augmented Generation). Those numbers come from multimodal tasks, so treat them as a sign of what planning can do rather than a promise for a text handbook. For the handbook, a one-time plan might be: keyword search "travel budget", then semantic search "definition of an international trip" in those results, then read the finance section on limits.
Checking keeps the plan on track. After each search, the grader asks whether the results address the sub-question; if not, the agent rewrites that sub-question or switches tools. If a search for "international definition" returns the password reset page, the grader rejects it and the agent tries "what counts as an international trip for travel policy". Surveys of agentic RAG call this a reflection pattern (Singh et al. 2025, Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG).
Why does agentic RAG avoid the failures of naive retrieval?
Naive retrieval has three weak spots that an agent addresses: blind trust, a fixed number of chunks, and a single search. Blind trust is assuming whatever the retriever returns is useful. Corrective RAG adds a lightweight evaluator that rates the retrieved documents for each question and triggers corrective actions when they look poor (Yan et al. 2024, Corrective Retrieval Augmented Generation). Agentic RAG puts a similar grading step between retrieval and generation, so weak results lead to a new search instead of an answer.
A fixed number of chunks hurts when a question needs more than the top k. A multi-part question about benefits might retrieve chunks on health insurance and miss the separate section on retirement contributions. An agent can notice the gap and search again specifically for retirement contributions before answering.
A single search struggles when the answer chains facts from two or more places. In the travel budget example, the approval hierarchy, the definition of an international trip and the dollar threshold might live in three sections. Interleaving retrieval with reasoning helps here: in the IRCoT study, alternating retrieval steps with chain-of-thought reasoning improved retrieval by up to 21 points and question answering by up to 15 points on four multi-step datasets (Trivedi et al. 2022, Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions).
- Retrieves the top k chunks once
- Can miss facts in other sections
- Answers from whatever came back
- Breaks the question into sub-questions
- Searches each part, and again if needed
- A grader checks before answering
What does it cost?
Every decision in the loop is a model call. A static pipeline calls the model once per question. The LangGraph loop above calls it at least twice, once to decide to search and once to answer, plus once to grade and once to rewrite on every pass through the loop. A hard question that needs two rewrites can take six or more calls, and each call adds waiting time. Grading each retrieved document separately, which some designs do, adds a call per document.
So budget for it. Set a maximum number of loops, measure the extra latency on real questions, and use a smaller, faster model for routing and grading if quality allows. Part 7's evaluation tells you whether the agent's better answers on hard questions are worth the extra cost on easy ones, which are usually most of them.
How do you implement agentic RAG for a company handbook assistant?
Start from what this series has built. Part 5's hybrid_search becomes the agent's main tool, Part 6's citation checks stay on the final answer, and Part 7's harness measures whether the agent beats the plain pipeline. In LangGraph you define a graph with nodes for deciding, retrieving, rewriting and answering, plus the grading function between them; the state carries the conversation and the retrieved documents.
Useful tools for a handbook agent include a search over chunks, a "read this whole section" tool for small-to-big reading (Part 2), and a search restricted by metadata, for example to one country's addenda. Keep the tool descriptions precise, because the model chooses tools by reading them; Tool Calling and MCP: How Agents Touch the World covers how to write them.
The grading step is the most important safety feature. It can be an LLM call that judges relevance, or Part 5's cross-encoder: if the best re-ranker score is below the threshold you calibrated in Part 6, treat the results as irrelevant and rewrite. Log every search and its grade. Over time, the failures tell you which tool descriptions, prompts or retrieval settings to improve.
The whole pipeline
Here is everything the series built, in the order a question flows through it.
| Stage | Where it was taught | File |
|---|---|---|
| Why retrieve at all, and when not to | Why RAG Exists: The Context Problem | none |
| Chunking by structure, in model tokens | Chunking: The Decision Everything Downstream Inherits | chunking.py |
| Embeddings and exact search | Embeddings and Vector Search, Demystified | embed.py |
| A vector database with permission filters | Vector Databases: What Actually Matters When Choosing | store.py |
| Keyword search, fusion and re-ranking | Hybrid Search and Re-Ranking: The Cheapest Quality Win | search.py |
| Citations you can check, and refusals | Grounding and Citations: Answers You Can Check | answer.py |
| Metrics and a regression gate | Evaluating RAG: Retrieval Metrics That Predict Answer Quality | evaluate.py |
| Agents that search again | This part |
If your questions are about relationships between things rather than passages of text, such as who reports to whom, RAG vs Knowledge Graphs: Two Kinds of AI Memory Lookup shows when a graph lookup beats vector search, and Graph Databases for AI: When Relationships Are the Data goes deeper.
Quick Reference
| Property | Value |
|---|---|
| Core control loop | Decide, retrieve, grade, rewrite if needed, retrieve again, answer |
| Retrieval tools in A-RAG | keyword_search, semantic_search, chunk_read |
| LangGraph tutorial | Nodes generate_query_or_respond, retrieve, rewrite_question, generate_answer; grade_documents routes between them |
| Grading | An LLM relevance check, or a re-ranker score against a calibrated threshold |
| Cost | At least two model calls per question, plus a grade and a rewrite per extra loop |
| Research results | A-RAG ahead on multi-hop benchmarks with GPT-5-mini, not always with GPT-4o-mini; E-Agent 13% more accurate with 37% fewer redundant searches on multimodal tasks |
Frequently Asked Questions
Q: How does agentic RAG handle multi-turn conversations where the user follows up on a previous answer?
The agent keeps the message history in its state. When a follow-up arrives, it can see what it already retrieved and avoid fetching the same documents again. If the user asks "Can you give me the details on that exception you mentioned?", the agent searches only for that detail.
Q: Does an agentic retriever need a separate model, or can the same LLM handle routing, grading and generation?
Many implementations use one LLM for every step, calling tools through function calling, and grade with the same model through a short classification prompt. Systems that care about speed and cost sometimes use a smaller model, or a cross-encoder, for routing and grading.
Q: How do you prevent an agentic RAG system from getting stuck in a retrieval loop?
Set a maximum number of loops, and let the grader accept "no good context" as a valid outcome. If the results keep failing the grade, the agent falls back to an honest answer such as "I couldn't find this in the handbook" instead of looping forever.
Q: Can agentic RAG work with hybrid search, or does it replace keyword and vector search?
It builds on them. The agent decides when to call which search, but you still need both the keyword and the vector index underneath; in the handbook assistant, the agent's main tool is Part 5's hybrid search.
Q: Is agentic RAG only useful for large, mixed document sets, or does it help with a single handbook too?
It can help with any collection where questions vary in difficulty: even one handbook has multi-part questions and vague ones. The grader catches many mismatches, but it is only as good as its instructions, as the test below shows. If most of your questions are simple lookups, measure first: the plain pipeline may be just as accurate, faster and cheaper.
Test yourself
A user asks your handbook assistant: "What is the cap on home-office equipment reimbursement for employees in Germany?" Your agentic system retrieves the global equipment policy chunk, which mentions no country-specific cap, and the grader rates it as relevant. The answer confidently states "There is no cap," but a German-specific addendum buried in a PDF actually sets a €500 limit. The user later reports the error. What went wrong in your agentic loop, and how would you fix it without abandoning the agentic pattern?
Answer: The grader judged the global policy relevant because it is about equipment reimbursement, but it had no idea that country rules might override it. Relevant is not the same as complete. Fix the grader's instructions: ask not only "is this relevant?" but "does this fully answer the question, including any location the user named?" Give the agent a tool that searches country addenda by metadata, and tell it to call that tool whenever a question names a country. Check that the German addendum was chunked and indexed at all, since a PDF that failed to parse would be invisible to every search. Finally, add this question and a few like it to Part 7's labelled set, so the fix is measured and a later change cannot quietly break it. The loop stays the same; the grader and the tools get better informed.
If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.
Sources
- What is agentic RAG? (IBM)
- Build a custom RAG agent with LangGraph (LangChain, read October 2026)
- A-RAG: Scaling Agentic Retrieval-Augmented Generation via Hierarchical Retrieval Interfaces (Du et al. 2026)
- Efficient Agent: Optimizing Planning Capability for Multimodal Retrieval Augmented Generation (Wang et al. 2025)
- Corrective Retrieval Augmented Generation (Yan et al. 2024)
- Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG (Singh et al. 2025)
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions (Trivedi et al. 2022)
