Evaluating Prompts: Beyond 'Looks Good to Me'
Test sets, A/B comparisons, and regression checks for prompt changes.
Prompt evaluation is the systematic measurement of how a prompt-model-configuration bundle performs on a defined task, using reproducible test sets and scoring methods instead of ad hoc inspection. It turns prompt iteration into a software testing discipline: golden datasets, automated metrics, LLM (large language model) judges, and A/B comparisons replace the “looks good to me” shrug.
LLM judges can be more consistent than human raters, but that consistency often masks a failure to capture real-world nuance. A judge that agrees with itself 99% of the time can still be wrong 20% of the time. Hybrid calibration, humans define the rubric, models scale it, is what separates a reliable evaluation pipeline from a false sense of security.
How do you build a test set that actually catches regressions?
A golden dataset is a versioned collection of inputs paired with ground-truth labels or rubrics. It serves as the regression suite for your prompt. If a prompt change degrades performance on this set, you catch it before it reaches users.
Start with real user tasks, not synthetic examples. For the customer-support reply assistant, pull actual tickets that span common issues, edge cases, and scenarios where the assistant previously failed. Include multi-turn conversations if the assistant maintains context. The goal is coverage of failure modes that matter, not sheer volume.
Ground truth can be a reference answer, a set of acceptable outputs, or a rubric describing what a correct response must contain. For the assistant, a rubric might specify that the reply must address the customer’s question, use a polite tone, and never hallucinate policy details. Statsig’s guidance on golden datasets emphasizes anchoring every test case to a user impact metric, like resolution accuracy or time saved.
Labeling requires domain expertise. Two labelers should independently judge outputs, resolve disagreements, and refine the rubric. This process estimates label noise and forces clarity. Autorubric’s framework formalizes this with reliability metrics like Cohen’s kappa, which quantifies agreement beyond chance. Autorubric
Maintain the dataset like source code. Version it. Add new cases when production logs reveal novel failures. Keep it tight. A bloated golden set slows iteration without improving signal. This dataset becomes the foundation for every evaluation that follows.
How does A/B testing for prompts work?
A/B testing for prompts compares two prompt variants on identical inputs and measures which one performs better on defined metrics. It can run offline on the golden dataset or online on live traffic.
Offline A/B tests are the first gate. You run both prompt variants against every test case in the golden set, score the outputs, and aggregate the results. The scoring layer can use deterministic checks (JSON (JavaScript Object Notation) schema validation, regex for forbidden patterns), semantic similarity, or LLM judges. Promptfoo and DeepEval both support this pattern: define a dataset, define metrics, run all variants, and view side-by-side comparisons. promptfoo DeepEval
For the customer-support assistant, you might compare a prompt that includes a detailed few-shot example against one that relies on a chain-of-thought instruction. The offline test would show which variant produces more factually correct replies and fewer policy hallucinations, as judged by an LLM rubric.
Online A/B testing routes a fraction of real user traffic to each variant and tracks production metrics: task success rate, user satisfaction score, latency, token cost. Platforms like Braintrust surface these as experiment results, with statistical significance computed automatically.
Shadow testing is a safer precursor: run the new prompt on a copy of live traffic without affecting users, then compare its outputs to the current prompt offline. Only when the offline and shadow results are clean do you proceed to a canary deployment. This staged rollout prevents a prompt change from silently degrading the assistant’s reliability in production.
What makes an LLM judge reliable?
An LLM judge is a model that reads a prompt’s output and scores it according to a rubric. Reliability means the judge’s scores correlate with human judgments and do not drift over time.
The judge prompt is the critical component. It must include the task description, the input, the output to evaluate, and the rubric criteria. G-Eval popularized chain-of-thought in judge prompts: the model reasons step by step before assigning a score, which improves alignment with human raters. G-Eval
Autorubric extends this with multi-judge ensembles and bias mitigations. It shuffles option order to reduce position bias, penalizes verbosity to counteract the tendency to rate longer outputs higher, and enforces per-criterion atomic evaluation so the judge does not conflate correctness with fluency.
The Judge’s Verdict Benchmark evaluates judges themselves. It found that many LLM judges correlate well with humans on average but are “super-consistent”: they agree with themselves far more than humans agree with each other. That super-consistency can hide systematic errors. A judge that always gives a 7/10 to a certain class of error is consistent but useless. Judge’s Verdict Benchmark
Calibration against human labels is non-negotiable. Run the judge on a subset of the golden dataset where humans have provided scores. Compute correlation and Cohen’s kappa. If the judge is super-consistent but misaligned, adjust the rubric or add few-shot examples in the judge prompt. Galileo’s analysis found that elite teams achieve 97-98% accuracy by front-loading human judgment into rubric design and using LLM judges only for scale, with ongoing validation.
For the assistant, a reliable judge would accurately flag replies that hallucinate return policies, even if the reply is fluent. That requires a rubric that explicitly penalizes unsupported policy claims and a calibration set where humans have marked such hallucinations.
How do you integrate prompt evaluation into CI?
Prompt evaluation becomes part of the CI pipeline when every prompt change triggers an automated eval run against the golden dataset. The run produces a report that gates the merge: if scores drop below a threshold, the change is blocked.
The pipeline looks like this:
The eval driver is a script that reads the dataset, iterates over prompt variants, calls the model, and applies scoring functions. OpenAI’s evals framework and promptfoo both provide CLI (command-line interface) tools that output JSON results, which CI can parse. OpenAI Evals
Caching is essential for speed and cost. Store generated outputs keyed by prompt hash and input. If a prompt variant hasn’t changed, reuse the cached output. Autorubric’s infrastructure supports resumable runs with checkpointing, which avoids re-invoking expensive judge models. Autorubric paper
Thresholds should be set based on historical variance. Run the same prompt multiple times to measure score noise, then set a threshold that is at least two standard deviations below the baseline mean. This prevents false alarms from stochasticity.
For the customer-support assistant, a CI eval might check that the new prompt does not increase hallucination rate by more than 2% and does not reduce correctness below 95%. If it does, the engineer gets a report showing which test cases regressed, with the judge’s explanations.
Why do naive evaluations fail in production?
Naive evaluation, spot-checking a few outputs, fails because it misses rare but critical failure modes. A prompt that works on 95% of cases can still produce a catastrophic hallucination on the 5% that matter most.
Prompt injection is a concrete example. An attacker can embed instructions in user input that override the system prompt. If your evaluation never includes adversarial inputs, you ship a prompt that is trivially exploitable. AgentFuzzer applies fuzzing techniques to discover such vulnerabilities automatically. AgentFuzzer
Another failure mode is model drift. When the underlying model is updated by the provider, the same prompt can behave differently. Without a regression suite, you discover the drift only when users complain. A golden dataset run on a schedule catches drift early.
Cost and latency regressions also hide in naive evals. A prompt that adds a chain-of-thought step might improve quality but double token usage. An A/B test that tracks cost per request alongside quality metrics reveals that trade-off. Without it, you optimize for quality and silently blow the budget.
The customer-support assistant is a good example. A naive evaluator might see that the new prompt writes friendlier replies and ship it. A systematic eval would catch that those friendlier replies sometimes invent refund policies, and that the prompt uses 30% more tokens. The difference is between a tool that helps users and one that creates liability.
- Spot-checks a few outputs
- Relies on subjective impression
- Misses rare failure modes
- No regression testing
- Runs full golden dataset
- Uses defined metrics and rubrics
- Catches rare failures like hallucination
- Automated regression suite
| Property | Value |
|---|---|
| Default eval dataset format | JSONL (one JSON object per line) |
| Common deterministic metrics | JSON schema compliance, regex match, exact match |
| Common LLM judge metrics | Correctness, groundedness, coherence, safety |
| Recommended judge model | GPT-4 or Claude 3.5, calibrated against human labels |
| Reliability metric for judges | Cohen’s kappa (target > 0.7) |
| CI integration pattern | CLI tool outputs JSON, parsed by CI to gate merges |
| Offline A/B test tool | promptfoo, DeepEval, OpenAI Evals |
| Online A/B test platform | Braintrust, Statsig |
Frequently Asked Questions
Q: How do I know if my golden dataset is representative? Check coverage of real failure modes by sampling production logs and comparing the distribution of input types. If the dataset misses an entire category of queries that users actually ask, it is not representative. Update it regularly with new cases from production.
Q: Can LLM judges replace human evaluation entirely? No. LLM judges are used for scale, but they must be calibrated against human labels on a representative subset. Without calibration, a judge can be consistently wrong. Humans define the rubric and validate the judge periodically.
Q: How do I handle non-deterministic outputs in regression testing? Run each test case multiple times and aggregate scores. Use statistical thresholds based on variance to decide if a score drop is real. Caching generated outputs can also stabilize results for a given prompt version.
Q: What is the cost of running evals at scale? It depends on the judge model and dataset size. Using GPT-4 as a judge on 1,000 test cases can cost tens of dollars per run. Caching, smaller judge models, and sampling can reduce cost. The cost of a production regression is usually far higher.
Q: How do I prevent prompt injection from corrupting my eval pipeline? Include adversarial inputs in the golden dataset that attempt to override the system prompt. Use fuzzing tools to generate injection payloads. The eval should measure whether the model follows the injected instruction instead of the intended task.
Test yourself
You are evaluating a new prompt for the customer-support assistant. The LLM judge reports a 10% improvement in correctness, but human spot-checks show no difference. What could be going wrong, and how would you investigate?
Answer: The judge may be biased toward the new prompt’s style, longer, more confident replies often score higher even when content is unchanged. First, check if the judge’s rubric penalizes verbosity or confidence. If not, add a length penalty and rerun. Second, compute the judge’s correlation with human labels on a calibration set. If correlation is low, the judge is not aligned; refine the rubric or add few-shot examples. Third, inspect the per-criterion breakdown. The judge might be inflating a sub-score like fluency while correctness is flat. Finally, run a blinded human evaluation on a larger sample to get a ground-truth estimate. If the human evaluation confirms no improvement, the judge’s signal is noise, and the rubric needs redesign.
If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.