Evaluating RAG: Retrieval Metrics That Predict Answer Quality
Recall@k, faithfulness, and the eval harness that catches regressions.
A RAG (retrieval-augmented generation) pipeline lives or dies on whether the retriever finds the right evidence. If the passage that answers a question never reaches the prompt, no model can recover it: it can only guess or refuse. That makes recall, did we retrieve what the answer needs, the first thing to measure. But more retrieved text is not free, because passages that look relevant and are not can still mislead the model. And even perfect retrieval does not guarantee a faithful answer.
This part defines the metrics that matter, explains which question each one answers, and builds the series' sixth file: an evaluation harness for the handbook assistant that compares retrieval methods and fails a build when quality drops.
Why does RAG need its own retrieval metrics?
Classical search metrics were designed for people scanning a results page. In RAG the reader of the results is an LLM (large language model), which reads every retrieved chunk in one pass. That changes what matters most.
Think of a librarian fetching pages for a writer. The writer can skip a few irrelevant pages, as long as every fact they need is somewhere in the stack. If one critical page is missing, the article cannot be written honestly. The same holds for the handbook assistant: if the retriever misses the remote work policy, the model will either refuse or invent one, however clean the rest of its context is.
So missing evidence is usually worse than extra noise. But noise is not harmless. Studies of RAG find that the passages that hurt most are the ones that look relevant without being so: retrieved passages that score highly but do not contain the answer degrade accuracy (Cuconasu et al. 2024, The Power of Noise: Redefining Retrieval for RAG Systems), and as more passages are added, answer quality first improves and then declines, with these "hard negatives" a key cause (Jin et al. 2024, Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG). Language models can also be easily distracted by irrelevant information placed in a problem (Shi et al. 2023, Large Language Models Can Be Easily Distracted by Irrelevant Context). That is why Part 5 re-ranks, and why you should measure rather than simply retrieve more.
A metric cheat sheet
Four ranking metrics come up again and again. Each is computed for one question, then averaged over a labelled set of questions, where a person has marked which chunks answer each one.
- Recall@k: the share of the relevant chunks that appear in the top k results. If a question needs two chunks and one is in the top 3, recall@3 is 0.5.
- Precision@k: the share of the top k results that are relevant. If one of the top 3 is relevant, precision@3 is 0.33.
- MRR, mean reciprocal rank: for each question, one divided by the position of the first relevant result (1 for first place, 0.5 for second), averaged over questions.
- nDCG@k, normalized discounted cumulative gain: credit for each relevant result, discounted by its position (divided by the logarithm, base 2, of its rank plus one), then divided by the best score a perfect ordering could get. 1.0 means a perfect order.
Careful: this recall@k is not the ANN recall from Part 3. ANN recall compares a fast index with an exact search. This recall compares the results with what people marked as relevant, and it is the one that predicts whether your answers can be right.
How do recall and precision relate to answer quality?
Recall tells you whether the evidence was there at all. If recall at your chosen depth is low, the model never sees what it needs, and no prompt can fix that. So measure recall first, at the depth you actually pass to the model.
Precision tells you how much else came along. It matters less than recall, but it is not irrelevant: more chunks cost tokens, push the important chunk toward the middle of a long prompt, and bring in passages that can mislead. Precision@k also has a ceiling you should know about: if a question has one relevant chunk and you retrieve three, precision@3 can never exceed 0.33, however good the system is.
The practical approach: pick a depth k that fits your prompt budget, measure recall@k on labelled questions, and then check answer accuracy at a few values of k. Use the smallest k that gives acceptable answers, and use ranking metrics such as MRR and nDCG to make sure the best chunk is near the top.
What are context precision and context recall?
Ragas, an open-source evaluation library for RAG, adds metrics that use an LLM as a judge, so you can evaluate without labelling every chunk by hand (Ragas documentation).
Context recall asks whether everything the perfect answer needed was retrieved. Ragas breaks a reference answer into claims and asks a judge whether each claim is supported by the retrieved chunks. Context precision asks whether the useful chunks were ranked first. Ragas asks a judge which retrieved chunks are relevant, then averages precision@k at each position where a relevant chunk appears. So it rewards putting the useful chunks first, unlike plain precision, which ignores order (Ragas context precision).
For your handbook assistant, a chunk about the vacation policy does not help a question about expense reports, even though both are HR documents. Context precision catches it when such chunks crowd the top of the list; context recall catches it when the clause about international travel was never retrieved.
- Recall, precision, MRR, nDCG
- Need human relevance labels
- Measure where the known relevant items rank
- Cannot see whether the answer used them
- Context recall, context precision, faithfulness
- The judge labels relevance on the fly
- Faithfulness checks the answer against the context
- Cheaper to scale, but the judge needs auditing
How does faithfulness evaluation complement retrieval metrics?
Faithfulness checks whether every claim in the generated answer is supported by the retrieved context. It catches generation failures that retrieval metrics cannot see: even with perfect recall, the model might misread a policy or combine facts wrongly.
Ragas computes faithfulness by breaking the answer into individual claims and asking a judge whether each can be inferred from the context; the score is the share of supported claims, from 0 to 1. DeepEval offers a similar metric (DeepEval's faithfulness metric). Read the two kinds of metric together. Low faithfulness with high context recall means the generator is at fault: the evidence was there and the answer strayed from it. Low faithfulness with low context recall points to retrieval as the root cause.
In the handbook assistant, a faithfulness check would catch "You can work remotely from any country" when the retrieved policy says remote work abroad needs approval and is limited to 20 days a year. Part 6's quote check catches the cruder version of this for free, on every answer.
Build it: an eval harness that catches regressions
An eval harness runs on every change to the pipeline. It takes a labelled set of questions, runs retrieval, computes the metrics, compares them with a baseline, and fails when they drop. This is the series' sixth file, evaluate.py. It needs no model calls: it compares the three retrieval methods from Parts 3 and 5 on ten labelled questions.
When we ran it (October 2026), it printed:
Keyword search missed a relevant chunk for one question ("How do I book time off?"), so its recall@3 is 0.9. Vector search found everything but ranked the HR-27 chunk second, so its MRR@3 is 0.95. Hybrid search with re-ranking put the right chunk first every time. Precision@3 stays near 0.33 for all three because nine of the ten questions have only one relevant chunk: the ceiling described above, not a flaw. Ten questions are far too few to trust for a real system, but the mechanics are the same at 200.
To see the gate work, point it at a worse retriever. When we swapped hybrid_search for keyword_search in the gate, it stopped with AssertionError: recall@3 fell to 0.9 (baseline 1.0). In a CI (continuous integration) job, that failed assertion blocks the merge.
Where do the labelled questions come from? Start with 30 to 50 real questions from users or support tickets, and mark the chunks that answer each one. To get more, generate questions from your documents with an LLM, Ragas includes a test set generator for this (Ragas test set generation), and have a person check every generated question before it joins the set. Label at the level of passages or facts where you can, so the labels survive a change of chunking (Part 2 explains why).
Adding judged metrics with Ragas
Faithfulness, context recall and answer relevance need a judge model, so they cost money and vary a little from run to run. The current Ragas API, version 0.4, scores one sample at a time with metrics from ragas.metrics.collections; its older evaluate() function still runs but is deprecated (Ragas migration guide). We checked this snippet against Ragas 0.4.3 installed in a throwaway environment, but did not run the judge call, because it needs an API key and a paid model. When we installed Ragas 0.4.3 in October 2026, it also needed langchain-community older than 0.4 to import.
The same module offers ContextRecall, ContextPrecision and AnswerRelevancy, each with its own ascore arguments. Because judge scores move a little between runs, compare them with a baseline using a tolerance, as evaluate.py does, rather than demanding an exact number.
Can you trust an LLM as the judge?
Often enough to be useful, if you check it. In the paper that introduced Ragas, its faithfulness judgments agreed with human annotators in 95 percent of pairwise comparisons on its test set, answer relevance in 78 percent and context relevance in 70 percent (Es et al. 2023, Ragas: Automated Evaluation of Retrieval Augmented Generation). For open-ended chat answers, strong judge models matched human preferences more than 80 percent of the time, about as often as humans agreed with each other (Zheng et al. 2023, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena). Those numbers come from their own test sets, not yours. Before you trust a threshold, have a person review 20 to 50 of the judge's decisions on your data.
Quick Reference
| Metric | What it measures | What it tells you |
|---|---|---|
| Recall@k | Share of the relevant chunks in the top k | Whether the evidence reached the model; measure this first |
| Precision@k | Share of the top k that are relevant | How much noise came along; capped when few chunks are relevant |
| MRR | Average of one over the rank of the first relevant chunk | Whether the best chunk comes first |
| nDCG@k | Relevance discounted by position, against a perfect order | The quality of the whole ranking |
| Context recall (Ragas) | Whether the reference answer's claims are supported by the retrieved chunks | Completeness of retrieval, judged by an LLM |
| Context precision (Ragas) | Average precision@k at the ranks of relevant chunks | Whether useful chunks are ranked first |
| Faithfulness | Share of the answer's claims supported by the context | Whether the generator stayed with its evidence |
| Answer relevance | Whether the answer addresses the question | Off-topic or evasive answers |
Test yourself
Your handbook assistant currently retrieves 5 chunks and achieves 0.80 recall@5. You increase top-K to 20. Recall@20 jumps to 0.95, but answer accuracy does not improve. What might be happening, and how would you diagnose it?
Answer: The extra chunks contain more of the evidence, but the model is not using it well. With 20 chunks, the passages that matter may sit in the middle of a long prompt, where models use information less reliably, and the extra chunks may include passages that look relevant but are not, which can mislead. The new evidence may also be redundant, repeating facts that were already there. To diagnose, compare answer accuracy with the first 5 chunks only against all 20; if it is the same, the later chunks are not helping. Check where the relevant chunks land in the 20, and read a sample of failed answers next to their contexts. The fix is usually to re-rank the 20 candidates and pass only the best few to the model, so the evidence stays near the top and the noise stays out.
Frequently Asked Questions
Q: Should I use Ragas's context precision if I already compute classical precision? They answer different questions. Classical precision@k ignores order and needs your labels. Ragas's context precision rewards ranking relevant chunks first and uses an LLM judge to decide relevance, so it can run without labels but needs auditing. If you have labels, MRR and nDCG already measure the ordering.
Q: How do I get ground-truth relevant chunks for context recall? Mark by hand which chunks answer a small set of real questions; this is the most valuable hour you will spend on evaluation. To scale, generate candidate questions with an LLM and have a person check each one, and record the passages each reference answer came from.
Q: Does faithfulness require a human-annotated answer? No. Faithfulness only needs the generated answer and the retrieved context. To evaluate whether the answer is correct, you still need a reference answer. Faithfulness alone tells you the answer is grounded in what was retrieved, not that it is right.
Q: How often should I run the full eval harness? On every pull request that touches retrieval, chunking or prompts, and at least nightly on the main branch. The retrieval metrics cost nothing to compute; the judged metrics cost a little per run, so you might run those nightly rather than on every change.
Next: Agentic RAG: When the Retriever Starts Thinking. It gives the model tools to decide what to search for, grade what it found, and search again.
If you want this kind of breakdown every week, how real RAG systems actually work under the hood, not just the blog-post summaries, subscribe to Internals Decoded at internalsdecoded.com.
Sources
- The Power of Noise: Redefining Retrieval for RAG Systems (Cuconasu et al. 2024)
- Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG (Jin et al. 2024)
- Large Language Models Can Be Easily Distracted by Irrelevant Context (Shi et al. 2023)
- Ragas documentation (read October 2026)
- Context precision (Ragas)
- Migration from v0.3 to v0.4 (Ragas)
- Test set generation (Ragas)
- Ragas: Automated Evaluation of Retrieval Augmented Generation (Es et al. 2023)
- Faithfulness metric (DeepEval)
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al. 2023)
