Evaluating RAG: Retrieval Metrics That Predict Answer Quality
Recall@k, faithfulness, and the eval harness that catches regressions.
A RAG (retrieval-augmented generation) pipeline lives or dies on whether the retriever finds the right evidence. The metrics that actually predict answer quality are recall under your token budget, context recall, and faithfulness. Precision and F1 can be misleading. An eval harness built around these metrics, using tools like RAGAS or DeepEval, catches regressions before they reach users.
Here is the counterintuitive part: In multi-hop RAG tasks, precision often drops as you retrieve more chunks, yet answer accuracy improves. F1 can even show a negative correlation with accuracy. The LLM (large language model) tolerates noise far better than it tolerates missing facts. Retrieval evaluation for RAG must center on recall, not on the balanced scores that search engines optimize.
Why does RAG need its own retrieval metrics?
RAG changes the game because the consumer of retrieval is an LLM, not a human scanning a search results page. An LLM can ignore irrelevant chunks if the relevant ones are present and near the top. It cannot fabricate evidence that was never retrieved. This asymmetry means recall dominates answer quality in a way that traditional IR metrics do not capture.
Think of a librarian fetching documents for a writer. The writer can skip irrelevant pages as long as all the necessary facts are somewhere in the stack. If even one critical page is missing, the writer cannot complete the article honestly. The librarian’s precision matters less than recall. The same logic applies to your company handbook assistant: if the retriever misses the policy on remote work, the LLM will either hallucinate or refuse to answer, no matter how clean the rest of the context is.
Classical IR metrics like precision, recall, F1, MRR, and NDCG were designed for ranking quality in search engines. They assume a human user who scans titles and snippets, where every irrelevant result wastes attention. In RAG, the LLM processes the entire context in one pass. Noise is cheap. Missing evidence is catastrophic. So we need metrics that measure whether the retriever provides enough evidence to answer the question, not just whether it ranks relevant documents highly.
How do recall and precision predict answer quality under a fixed token budget?
Recall strongly predicts answer accuracy. Precision does not. Under a fixed context token budget, increasing recall almost always improves the LLM’s ability to answer correctly, even when it adds irrelevant chunks. The multi-hop QA (quality assurance) study by Li et al. shows recall correlates positively with accuracy across datasets like ClapNQ and MuSiQue, while precision decreases as context grows and F1 can correlate negatively source.
This happens because modern LLMs are robust to irrelevant text. They attend to the parts that matter. When you give the model more chunks, you increase the chance that all required evidence appears somewhere in the prompt. The extra noise does not confuse the model enough to offset the gain from having the right facts. Precision drops because the proportion of relevant chunks shrinks, but answer quality rises.
The practical takeaway: optimize recall at your chosen top-K, not F1. If you have a 4,000-token budget, measure recall@k where k is the maximum number of chunks that fit. Plot recall@k against answer accuracy. You will likely see a monotonic relationship. Use that curve to pick the smallest k that gives acceptable accuracy, not the k that maximizes a balanced IR metric.
What are context precision and context recall, and how do they differ from classical metrics?
Context precision measures how much of the retrieved context is actually useful for answering the question. Context recall measures whether all the information needed to answer was retrieved. Unlike document-level precision and recall, these metrics are tied to the question and, for context recall, to the ground-truth answer.
RAGAS computes context precision by asking an LLM judge to label each retrieved chunk as relevant or not to the question, then taking the fraction of relevant chunks. This is a signal-to-noise ratio specific to the query. Context recall is computed by extracting claims from the ground-truth answer and checking whether each claim is supported by any retrieved chunk. It answers the question: “Did we retrieve everything the perfect answer needed?”
These metrics are more predictive than raw IR recall because they account for partial relevance and for the fact that some retrieved chunks may be topically related but not actually contain the answer. For your handbook assistant, a chunk about the vacation policy is not helpful for a question about expense reports, even if both are HR documents. Context precision catches that. Context recall ensures you did not miss the specific clause about international travel reimbursement.
- Precision, Recall, F1, MRR, NDCG
- Assume human scans results
- Measure ranking quality
- Treat all retrieved items equally
- Context Precision, Context Recall, Faithfulness
- LLM judge evaluates relevance
- Account for partial relevance
- Check answer consistency and support
How does faithfulness evaluation complement retrieval metrics?
Faithfulness checks whether every claim in the generated answer is supported by the retrieved context. It catches generation failures that retrieval metrics cannot see. Even with perfect recall, the LLM might hallucinate, misread a policy, or combine facts incorrectly. Faithfulness flags those errors.
RAGAS computes faithfulness by breaking the answer into atomic claims and asking an LLM judge whether each claim can be inferred from the context. DeepEval offers a similar metric source. The output is a score between 0 and 1. A low faithfulness score with high context recall tells you the generator is the problem. A low faithfulness score with low context recall tells you retrieval is the root cause.
In the handbook assistant, a faithfulness check would catch an answer like “You can work from anywhere in the world” when the retrieved policy only mentions domestic remote work. The retrieval step might have pulled the correct policy chunk, but the LLM overgeneralized. Without faithfulness, you would never know the answer was wrong.
What does an eval harness that catches regressions look like?
An eval harness runs on every pipeline change. It takes a labeled dataset of questions, gold answers, and gold chunk IDs, runs the full RAG pipeline, and computes a set of retrieval and generation metrics. It then compares the scores to a baseline and alerts on drops.
A minimal harness for the handbook assistant might look like this:
- Dataset: 200 question-answer pairs with annotated relevant chunks from the handbook.
- Retrieval metrics: recall@10, context recall (via RAGAS), context precision.
- Generation metrics: faithfulness, answer relevancy.
- Thresholds: recall@10 must stay above 0.85. Faithfulness must stay above 0.90. A drop in either triggers a CI failure.
You can implement this with RAGAS and a few dozen lines of Python. The harness runs after every change to the chunking strategy, embedding model, or retrieval pipeline. It gives you a clear signal: did this change break answer quality? Without it, you are flying blind.
Quick Reference
| Metric | What it measures | Predictive of answer quality? |
|---|---|---|
| Recall@k | Fraction of relevant chunks retrieved in top-k | Strongly yes, under fixed token budget |
| Precision@k | Fraction of retrieved chunks that are relevant | Weakly, can be misleading |
| F1@k | Harmonic mean of precision and recall | Unreliable, can correlate negatively |
| Context recall | Whether all evidence needed for the answer was retrieved | Yes, directly tied to answer completeness |
| Context precision | Signal-to-noise ratio of retrieved context | Moderately, helps diagnose noise |
| Faithfulness | Whether answer claims are supported by context | Yes, catches hallucinations |
| Answer relevancy | Whether the answer addresses the question | Yes, catches off-topic responses |
Test yourself
Your handbook assistant currently retrieves 5 chunks and achieves 0.80 recall@5. You increase top-K to 20. Recall@20 jumps to 0.95, but answer accuracy does not improve. What might be happening, and how would you diagnose it?
Answer: The extra chunks likely contain relevant evidence, but the LLM is not using it effectively. Possible causes: the generator’s effective context window is smaller than the total prompt length, so later chunks are ignored. Or the model’s attention fades after the first few chunks, a known issue with some architectures. You can test this by truncating the prompt to only the first 5 chunks and comparing accuracy. If accuracy stays the same, the later chunks are not being used. Another possibility: the additional evidence is redundant, not adding new information needed to answer. You can check this by measuring the novelty of each chunk relative to the answer. If the new chunks merely restate facts already present, recall improvement is hollow. The fix might be to re-rank so the most critical evidence appears first, or to reduce top-K and invest tokens in a re-ranker instead.
Frequently Asked Questions
Q: Should I use RAGAS’s context precision if I already compute classical precision? Context precision is query-aware and uses an LLM judge, so it catches cases where a chunk is topically related but not useful. Classical precision treats all relevant chunks equally. Use context precision when you need to detect noisy retrieval that classical metrics miss.
Q: How do I get ground-truth relevant chunks for context recall? You can manually annotate a small set of queries with the chunk IDs that contain the answer. For scale, you can use an LLM to generate candidate chunks and then have a human verify. Some teams use the chunks that originally supported the gold answer in their dataset.
Q: Does faithfulness require a human-annotated answer? No. Faithfulness only needs the generated answer and the retrieved context. It checks internal consistency. However, to evaluate answer correctness you still need a reference answer. Faithfulness alone tells you the answer is grounded, not that it is right.
Q: Can I trust LLM-as-a-judge for these metrics? Yes, with calibration. Studies show that strong LLMs like GPT-4 agree with human judgments on faithfulness and context relevance over 80% of the time. For high-stakes pipelines, you should periodically audit the judge’s decisions against human labels.
Q: How often should I run the full eval harness? On every pull request that touches retrieval, chunking, or prompting. At minimum, nightly on the main branch. The cost is low compared to shipping a regression that silently degrades answer quality for days.
If you want this kind of breakdown every week, how real RAG systems actually work under the hood, not just the blog-post summaries, subscribe to Internals Decoded at internalsdecoded.com.
Sources
- Recall, Context Length, and Efficient Multi-Hop Reasoning (Li et al.)
- DeepEval Metrics
- docs.confident-ai.com · Metrics Faithfulness