Evaluating Agents: Golden Sets, Judges, and Shadow Mode
Same input, different outputs. How to measure whether a nondeterministic system actually works.
A reliable agent evaluation pipeline rests on three pillars: a golden dataset of known-correct examples, a judge that can scale beyond human review, and a shadow mode that mirrors production traffic without user impact. Together they turn a nondeterministic system into something you can measure, compare, and improve with confidence.
Traditional software testing expects the same answer every time you press the button. AI agents don’t work that way. The same inbox can produce different urgency flags on two consecutive runs. The interesting question is not whether the output changed. It’s whether the change made things better or worse. And you can’t answer that question with a pass/fail test case.
Why standard test cases break with agents
A unit test for Mailmind’s email classifier looks simple on paper: feed it an urgent thread and assert “urgent” is returned. In practice, the classifier might say “urgent” 90% of the time and “routine” 10% of the time because it took a different reasoning path. That 10% fail rate on a deterministic assertion is noise, not a bug. If you gate deployments on 100% pass rates, you never ship.
The core challenge is that agent quality is judgment-based, not binary. Two equally reasonable responses to “Is this email urgent?” might differ. Traditional software testing cannot grade gradations of quality across multiple runs. Evaluation for agents needs to capture precision, recall, consistency, and safety, all while accepting that output variance is normal.
What exactly should you measure?
Direct answer: You need metrics across six axes: correctness, completeness, consistency, performance, reliability, and safety. No single number captures agent quality. Skipping one axis hides failures until users find them first.
Correctness measures whether the agent’s decisions match the ground truth. For Mailmind, it answers: did the system identify the correct set of urgent emails? Correctness splits into precision (of the emails flagged as urgent, how many truly are) and recall (of the true urgent emails, how many did we catch).
Completeness matters when the agent must extract structured information. Did Mailmind capture the meeting time, the refund amount, and the tracking number? A missed detail is as damaging as an incorrect one.
Consistency tracks how much the output changes across identical inputs. A high variance in classification without a change in the underlying model or data is a red flag for nondeterminism that has crossed from acceptable to problematic.
Performance covers latency and cost. For Mailmind, p95 latency on triaging a batch of 20 emails and the dollar cost per email are operational boundaries you need to monitor.
Reliability tracks failure modes: the rate at which tool calls fail, the model returns a malformed output, or the agent loop hangs. It’s the uptime metric for an AI component.
Safety measures guardrail triggers: prompt injection detections, out-of-scope tool authorization denials, and personally identifiable information leaks. Each trigger is a signal you evaluate as part of the system’s health.
How does a golden dataset anchor evaluation?
Direct answer: A golden dataset is a curated collection of inputs paired with expert-verified correct outputs and acceptable variance ranges. It is the source of truth that every other evaluation method calibrates against.
For Mailmind, a golden dataset starts with a hundred real inbox snapshots. Each snapshot has a set of emails. A human annotator labels which emails are urgent, what the correct extracted details are (dates, amounts, entities), and what the ideal drafted reply should contain. The annotation process is iterative: two annotators label independently, then reconcile differences with a third reviewer. The final set becomes the ground truth.
The dataset must be diverse enough to represent the real distribution. If 90% of the original emails are routine newsletters, your golden set needs to reflect that. But you also need enough examples from each edge case: refund requests with missing order numbers, meeting invites with ambiguous times, and emails that look urgent but are spam. A minimum meaningful size is typically a few hundred examples per intent or category, with a deliberate sampling strategy that over-samples failure modes discovered in production.
Once the golden dataset exists, you compute precision, recall, and F1 against it. Precision tells you how many of the emails Mailmind flagged as urgent were actually urgent. Recall tells you how many of the true urgent emails were caught. Together they give you an accuracy snapshot that respects the class imbalance between urgent and routine.
Updating the golden dataset is a continuous process. When a user reports that Mailmind missed an urgent email, that email gets anonymized, reviewed, and added to the set. When the system’s behavior drifts in a new direction (for example, after a domain expansion to include travel confirmations), you add representative samples for the new category. A stale golden dataset is worse than no dataset because it gives a false sense of safety.
How does LLM‑as‑judge scale evaluation?
Direct answer: A stronger language model grades agent outputs against a rubric. This method is fast, reusable, and can run on every commit. It must be validated against human judgment on the same golden dataset before you trust it.
In Mailmind’s pipeline, when the agent extracts a meeting time from an email, the judge model receives the original email, the extracted value, and a rubric that asks: “Is the extracted time exactly correct, partially correct, or wrong? Consider time zone ambiguity.” The judge outputs a structured grade. For classification tasks, the judge can compare the agent’s urgency label against the ground truth label and explain why it agrees or disagrees.
The judge is only as useful as its own agreement with human evaluators. You validate it by running both human reviewers and the judge on the same 200 golden examples. If the judge’s overall accuracy relative to the human consensus is above 95%, you can rely on it for automated regression testing. Below that threshold, you either improve the rubric, switch to a more capable judge model, or accept that certain tasks still require human spot checks.
One common failure mode: the judge rewards style over substance. An agent might produce a polite, confident but slightly incorrect draft reply, and the judge scores it highly because the rubric overweights tone. Tuning the rubric to penalize factual errors heavily (for example, a point deduction for each misstated date) is essential.
Where does human evaluation still fit?
Direct answer: Human evaluation is the gold standard that validates the golden dataset, calibrates the LLM judge, and audits production samples. It is too expensive to run on every change but is the only way to catch subtle judgment gaps.
In the Mailmind cycle, human reviewers perform three roles. First, they annotate the initial golden dataset, which is the foundation of everything downstream. Second, they spot-check a random sample of production outputs weekly, looking for issues that automated metrics miss, like a reply that is technically correct but socially off-key. Third, when the LLM judge flags an output as ambiguous, a human resolves the tie. This hybrid approach keeps quality high while controlling cost.
How does regression testing prevent silent breakage?
Direct answer: A regression suite re-runs the entire golden dataset and judge pipeline on every prompt, tool, or threshold change. Its goal is to detect the moment an improvement for one scenario degrades performance on another.
In Mailmind’s CI/CD pipeline, when an engineer updates the urgency classification prompt to catch more refund-related emails, the regression suite runs 300 golden examples through the updated agent. The outputs are scored by the LLM judge. If recall for refunds improves from 88% to 94% but precision on routine newsletters drops from 97% to 91%, the suite surfaces that tradeoff. The team decides whether the recall gain justifies the precision loss before merging.
The suite must run fast enough to integrate into a pull request workflow. Parallel execution spreads the load: 300 examples across 10 workers run in a couple of minutes. The key is that the suite catches regressions before a human reviewer ever sees the change.
What is shadow mode and why does it reduce deployment risk?
Direct answer: Shadow mode runs the new agent version in parallel with the existing one on live production traffic. Its outputs are logged and compared but never shown to users. This gives you a real-data risk preview without any user-facing impact.
For Mailmind, deploying a new reply-drafting agent is high stakes. Before flipping the switch, the team puts the new version in shadow mode for a week. Every incoming email triggers both the old and new agents. The outputs are compared automatically: how often do they disagree? When they disagree, does the new version’s output score higher under the judge? The team also monitors cost and latency under real load. If the new agent’s latency spikes at p95 under morning email volume, that’s caught before users experience it.
The comparison logic must not rely on exact string matching. Two different drafts can both be correct. Instead, the judge scores each output against the email context, and the team looks at the distribution of score differences. If the new agent consistently scores higher, the confidence to promote it grows. If scores are tied but the new agent is cheaper, that’s also a win.
How do you structure the evaluation pipeline end to end?
Direct answer: The pipeline progresses through five stages: unit, integration, regression, shadow, and production monitoring. Each stage adds realism and cost, and filters the change before it reaches the next.
- Unit stage: each tool and isolated agent function (extract date, classify urgency) runs against a small set of handcrafted examples. This catches obvious logic errors immediately.
- Integration stage: the full Mailmind agent pipeline runs on the golden dataset. The LLM judge scores outputs. This stage answers the question “Does the whole system work on known-good data?”
- Regression stage: every pull request runs the integration suite. This catches regressions when a change unintentionally impacts another part of the system.
- Shadow stage: the new version runs live without user impact. Metrics are compared against the production baseline.
- Production monitoring: after promotion, live dashboards track accuracy, latency, cost, guardrail triggers, and human-review rate. Alerts fire when metrics drift.
Each stage is a gate. If the regression suite shows a drop in recall for an important category, the change does not advance to shadow. If shadow mode reveals a latency regression, promotion halts until it’s fixed.
| Property | Value / Range |
|---|---|
| Golden dataset minimum size per intent | 200–500 examples |
| Judge agreement threshold vs. human | > 95% for automated deployment gating |
| Shadow mode comparison window | 3–7 days of real traffic |
| Key production metrics | Completion rate, p95 latency, cost/task, guardrail trigger rate |
| Recommended per-category tracking | Precision, recall, F1, human-review rate per email intent |
| Regression suite runtime target | < 5 minutes for PR feedback |
Frequently Asked Questions
Q: How do you know when your golden dataset is large enough? The size is sufficient when adding 50 new examples does not change precision or recall by more than 2 percentage points. Monitor this saturation point as you expand the dataset monthly.
Q: Can LLM‑as‑judge replace human evaluation entirely? No. Even with high agreement, the judge can develop systemic blind spots (e.g., over-prioritizing politeness). Humans must audit production samples and recalibrate the judge quarterly. The judge handles scale; humans handle calibration.
Q: What is the biggest mistake teams make with shadow mode? They assume identical outputs mean correct outputs. The new and old agent may agree on a wrong answer. Always compare outputs against ground truth scoring, not just each other.
Q: How do you prevent metric drift after deployment? Track per-category precision, recall, and human-review rates on a dashboard. Set thresholds that alert when the review rate rises week-over-week, signaling a slow degradation that aggregate accuracy hides.
Q: When does it make sense to skip shadow mode? Only for low-risk changes to non-decision-making components, like a caching layer update. Any change that alters the agent’s reasoning, tool calls, or output surface should run through shadow mode. The cost of a bad output hitting a real inbox is higher than the week of shadow time.
Test yourself
After a prompt update designed to improve recall of refund requests, Mailmind users complain that their routine newsletters are now being flagged as urgent. Your dashboard shows overall accuracy at 96%. The golden dataset reports 97% precision and recall. No regression was caught. Walk through what likely happened and how you fix it.
Answer: The golden dataset under‑represented routine newsletters. A prompt change that boosted refund sensitivity inadvertently lowered the classification threshold for anything with a dollar sign, catching newsletters that mention “free shipping worth $10”. The fix has two phases. First, roll back the prompt update to stop the user pain immediately. Second, expand the evaluation surface: add a false‑positive rate metric per category, and build a set of 50 annotated routine emails that include dollar amounts. Re‑run the old and new versions against this expanded set to isolate the exact few‑shot example or instruction that caused the over‑sensitization. Then tune the prompt to preserve the refund recall gain while restoring routine precision. Before redeploying, confirm that the updated regression suite shows both metrics within acceptable thresholds.
If you want this kind of breakdown every week, the real internals of production AI systems, subscribe to Internals Decoded at internalsdecoded.com.