Evaluating Prompts: Beyond 'Looks Good to Me'

Prompt evaluation is the systematic measurement of how a prompt, a model and its settings perform together on a defined task, using reproducible test sets and scoring instead of reading a few outputs. It turns prompt iteration into a software testing discipline: golden datasets, automated checks, LLM judges and A/B comparisons replace the "looks good to me" shrug.

This is the part of the series that answers the question the earlier parts kept asking: how would you know? Part 1 claimed that moving a rule to the start and end of a prompt helps; part 5 claimed that an uncertainty rule reduces invented policies. Below, you will build a small harness that tests exactly that kind of claim.

LLM judges can be more consistent than human raters, but consistency is not the same as being right. For example, a judge could give the same score 99 times out of 100 and still disagree with humans on 1 case in 5. Hybrid calibration (humans define the rubric and check the judge, models apply it at scale) is what separates a reliable evaluation pipeline from a false sense of security.

How do you build a test set that actually catches regressions?

A golden dataset is a versioned collection of inputs paired with ground truth labels or rubrics. It serves as the regression suite for your prompt. If a prompt change makes things worse on this set, you catch it before it reaches users.

Start with real user tasks, not synthetic examples. For the customer support reply assistant, pull actual tickets that span common issues, edge cases, and situations where the assistant failed before. Include multi turn conversations if the assistant keeps context. The goal is coverage of the failures that matter, not sheer volume.

Ground truth can be a reference answer, a set of acceptable outputs, or a rubric describing what a correct response must contain. For the assistant, a rubric might say the reply must address the customer's question, use a polite tone, and never invent policy details. Tie each test case to an outcome users care about, such as whether the issue was resolved.

Labeling needs people who know the domain. Two labelers should judge outputs independently, resolve their disagreements, and refine the rubric. This estimates how noisy your labels are and forces clarity. Agreement is usually measured with Cohen's kappa, a score of agreement that discounts the agreement two raters would reach by chance: 1 means perfect agreement, and 0 means no better than chance. Autorubric, an open source framework for rubric based evaluation, reports it among its reliability metrics (Autorubric: A Unifying Framework for Rubric-Based LLM Evaluation on Non-Verifiable Tasks, Rao and Callison-Burch).

Maintain the dataset like source code. Version it. Add new cases when production logs reveal new failures. Keep it tight: a bloated golden set slows iteration without adding signal. This dataset is the foundation for every evaluation that follows.

How does A/B testing for prompts work?

A/B testing for prompts compares two prompt variants on identical inputs and measures which one does better on defined metrics. It can run offline on the golden dataset or online on live traffic.

Offline A/B tests are the first gate. You run both variants on every test case in the golden set, score the outputs, and compare. Scoring can use exact checks in code (JSON Schema validation, a regular expression for forbidden phrases), semantic similarity, or LLM judges. Open source tools such as promptfoo and DeepEval package this pattern: define a dataset, define metrics, run all variants, and compare the results side by side.

Online A/B testing routes a fraction of real user traffic to each variant and tracks production metrics: task success rate, user satisfaction, latency and token cost. Shadow testing is a safer step before it: run the new prompt on a copy of live traffic without affecting users, then compare its outputs with the current prompt's offline. Only when the offline and shadow results are clean do you move to a canary deployment, a small share of real traffic. This staged rollout stops a prompt change from quietly making the assistant less reliable.

A worked example you can run

Here is the whole loop at a size you can read in one sitting. The question is part 1's: does moving the refund policy link rule from the middle of the system prompt to its start and end make replies include the link more often?

First, the golden set, one JSON object per line in a file called golden.jsonl. These are the first three of 20; in practice you would write 20 to 50 from real tickets:

{"id": "t01", "email": "My blender arrived with a cracked jug. I want my money back."}
{"id": "t02", "email": "Where is order 48213? It was due on Monday."}
{"id": "t03", "email": "You charged me twice for the same order. Fix this now."}

Next, the harness. It uses only Python's standard library. The check function holds the exact checks code can do, compare runs both prompts on every case and counts where they disagree, and sign_test asks how likely that split of disagreements would be if the prompts were really equal.

import json
import math
import random

LINK = "https://acme.example/refunds"
RULE = f"Always include the refund policy link ({LINK})."
GUIDELINES = "\n".join(f"Guideline {i}: keep the customer informed at every step." for i in range(1, 41))
PROMPT_A = f"You are a support agent for Acme Corp.\n{GUIDELINES[:1100]}\n{RULE}\n{GUIDELINES[1100:]}"
PROMPT_B = f"You are a support agent for Acme Corp. {RULE}\n{GUIDELINES}\nReminder: {RULE}"

def check(reply):
    """Code checks: exact, cheap, and the first thing to automate."""
    return {
        "policy link": LINK in reply,
        "under 150 words": len(reply.split()) <= 150,
        "no competitor": "globex" not in reply.lower(),
    }

def stand_in_model(prompt, email, rng):
    """NOT a real model: a stand-in so this file runs without an API key.
    It adds the policy link 90% of the time when the link rule sits within 200
    characters of the start or end of the prompt, and 60% of the time otherwise.
    We picked those numbers for the demo; nothing here is measured."""
    at = prompt.find("refund policy link")
    near_edge = at < 200 or at > len(prompt) - 200
    reply = "Hello, thanks for getting in touch. We are looking into this now."
    if rng.random() < (0.9 if near_edge else 0.6):
        reply += f" You can read our refund policy here: {LINK}"
    return reply

def evaluate(prompt, cases, model, seed):
    rng = random.Random(seed)
    return [check(model(prompt, case["email"], rng)) for case in cases]

def sign_test(a_only, b_only):
    """Two-sided exact sign test on the cases where the two prompts disagree."""
    n, k = a_only + b_only, max(a_only, b_only)
    if n == 0:
        return 1.0
    return min(1.0, 2 * sum(math.comb(n, i) for i in range(k, n + 1)) / 2 ** n)

def compare(prompt_a, prompt_b, cases, model, seed=1):
    a = evaluate(prompt_a, cases, model, seed)
    b = evaluate(prompt_b, cases, model, seed + 100)
    for name in a[0]:
        a_only = sum(x[name] and not y[name] for x, y in zip(a, b))
        b_only = sum(y[name] and not x[name] for x, y in zip(a, b))
        print(f"{name:16} A {sum(x[name] for x in a):2}/{len(a)}  B {sum(y[name] for y in b):2}/{len(b)}"
              f"  A only {a_only}  B only {b_only}  p = {sign_test(a_only, b_only):.2f}")

if __name__ == "__main__":
    cases = [json.loads(line) for line in open("golden.jsonl", encoding="utf-8")]
    compare(PROMPT_A, PROMPT_B, cases, stand_in_model)
    runs = [sum(r["policy link"] for r in evaluate(PROMPT_A, cases, stand_in_model, s)) for s in range(5)]
    print("prompt A, policy link, five runs:", runs)

The stand_in_model is the honest part to notice. It is not a model. We wrote it to follow the rule more often when the rule sits near the edges of the prompt, so that the harness has something to measure without an API key or a bill. Running the file with Python 3.13 and our 20 cases printed this (the stand-in ignores the email text, so any 20 cases give the same counts):

These numbers describe the stand-in, not any real model. What they teach is how noisy a small test set is. The prompts disagreed on 10 tickets: B won 8 and A won 2. The sign test gives p = 0.11, which means a split at least that lopsided would turn up about 11% of the time even if the two prompts were equally good. That is suggestive, not convincing, even though we built the stand-in so that B really is better. And rerunning prompt A five times with different random seeds gave 10, 11, 10, 11 and 13 links out of 20, so a difference of two or three tickets between prompts can be pure chance. With 20 cases, rerun several times or add cases before you trust a small difference.

To test a real model, replace the stand-in with a function that calls one, for example with OpenAI's Python SDK:

from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY from the environment

def real_model(prompt, email, rng=None):
    response = client.responses.create(model="gpt-6-astra", instructions=prompt, input=email)
    return response.output_text

and pass real_model to compare. Code checks cover what code can see. For what it cannot, such as whether the reply actually addresses the customer's problem, you need a judge.

What makes an LLM judge reliable?

An LLM judge is a model that reads an output and scores it against a rubric. A judge is reliable when its scores agree with human judgments and stay stable over time.

The judge prompt is the critical part. It must include the task, the input, the output to evaluate and the rubric. G-Eval popularized asking the judge to reason step by step before it fills in its scores, which improved its agreement with human raters (G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment, Liu et al.). A short rubric for our assistant, scoring one criterion at a time, might look like this:

You are grading a customer support reply. Use only the policy text below.
Answer each question with PASS or FAIL and one sentence of evidence.
1. Does the reply address the customer's actual issue?
2. Does every policy statement in the reply match the policy text?
3. Is the tone polite and calm, even if the customer is angry?
Return JSON: {"addresses_issue": ..., "policy_faithful": ..., "tone": ..., "evidence": [...]}

Ask for that JSON with structured outputs from Structured Outputs: JSON You Can Actually Parse, so your harness can count the verdicts like any other check.

Autorubric adds ensembles of judges and bias mitigations. It shuffles the order of answer options to reduce position bias, scores each criterion separately so the judge does not mix up correctness with fluency, and lets you give criteria negative weights, for example to penalize padding. Its authors note that it does not yet correct for verbosity bias, judges' habit of favoring longer answers, so watch length yourself (Rao and Callison-Burch).

Judges can be evaluated too. The Judge's Verdict benchmark first filters judges by how well their scores correlate with human labels, then uses Cohen's kappa and z-scores to compare their agreement with the variation between humans. Of 54 models, 27 reached its top tier, and four were "super-consistent", agreeing more than humans do with each other. The authors note this could reflect either better reliability or oversimplified judgments, so consistency alone does not prove good judgment (Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreement, Han et al.).

Calibration against human labels is not optional. Run the judge on part of the golden set where people have already scored the outputs, and compute the correlation and Cohen's kappa between the judge and the humans. If the judge is consistent but misaligned, adjust the rubric or add examples to the judge prompt, then check again. Front-load human judgment into the rubric and use the judge for scale.

For the assistant, a reliable judge would flag replies that invent a return policy, even when the reply reads well. That takes a rubric that explicitly fails unsupported policy claims and a calibration set where people have marked such inventions.

How do you integrate prompt evaluation into CI?

CI (continuous integration) is the set of automatic checks that run on every code change. Prompt evaluation becomes part of it when every prompt change triggers an evaluation run against the golden dataset. The run produces a report that gates the merge: if scores fall below a threshold, the change is blocked.

The pipeline looks like this:

The evaluation driver is a script like the one above: it reads the dataset, runs each prompt variant, calls the model and applies the checks. OpenAI's open source evals framework and promptfoo both offer command line tools that write their results as JSON, which CI can read.

Caching saves time and money. Store generated outputs keyed by a hash of the prompt and the input, and reuse them when a variant has not changed. Autorubric, for example, caches responses and can resume an interrupted run from a checkpoint instead of calling expensive judge models again.

Set thresholds from measured noise. Run the same prompt several times, as the five runs above did, to see how much the scores move on their own, then set a threshold well outside that range, for example two standard deviations below the baseline mean. That stops random variation from raising false alarms.

For the customer support assistant, a CI check might require that the rate of invented policies rises by no more than 2 percentage points and correctness falls by no more than 5 points from the current prompt's baseline. If either limit is crossed, the engineer gets a report of the test cases that got worse, with the judge's explanations.

Why do naive evaluations fail in production?

Naive evaluation, spot checking a few outputs, fails because it misses rare but critical failures. A prompt that works on 95% of cases can still produce a damaging invention on the 5% that matter most.

Adversarial inputs are a concrete example. If your test set never includes text that tries to override your instructions, you can ship a prompt that is easy to exploit, which Prompt Injection: The Attack Every AI App Inherits explains in depth. Automated red teaming tools now search for such attacks: AgentVigil, a fuzzing framework for indirect prompt injection against agents, reported attack success rates of 71% and 70% against agents built on o3-mini and GPT-4o on two public benchmarks, nearly double the baseline attacks (AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents, Wang et al.). Add a handful of injection attempts to your golden set and check that the assistant ignores them.

Another failure is model drift. When the provider updates the underlying model, the same prompt can behave differently. Without a regression suite, you find out only when users complain. A golden set run on a schedule catches drift early.

Cost and latency regressions also hide from naive evaluation. A prompt that adds a reasoning step might improve quality but double token use. An A/B test that tracks cost per request alongside quality makes that trade-off visible.

Picture the support assistant once more. A naive reviewer sees that a new prompt writes friendlier replies and ships it. A systematic evaluation would catch that those friendlier replies sometimes invent refund policies and use more tokens. That is the difference between a tool that helps users and one that creates liability.

Quick Reference

Frequently Asked Questions

Q: How do I know if my golden dataset is representative? Check its coverage of real failures by sampling production logs and comparing the kinds of input. If the dataset misses a whole category of questions users actually ask, it is not representative. Update it regularly with new cases from production.

Q: Can LLM judges replace human evaluation entirely? No. LLM judges give you scale, but they must be calibrated against human labels on a representative subset. Without calibration, a judge can be consistently wrong. People define the rubric and check the judge from time to time.

Q: How do I handle non-deterministic outputs in regression testing? Run each test case several times and look at the spread, as in the worked example. Use thresholds based on that spread to decide whether a drop is real. Caching generated outputs keeps results stable for a given prompt version.

Q: What is the cost of running evals at scale? It depends on the judge model and the size of the dataset. Running a frontier judge model on 1,000 test cases costs real money on every run. Caching, smaller judge models and sampling cut the cost, and the cost of a production regression is usually far higher.

Q: How do I prevent prompt injection from corrupting my eval pipeline? Include adversarial inputs in the golden set that try to override the system prompt, and measure whether the model follows the injected instruction instead of its task. Treat the outputs you feed to a judge as data too: a reply that contains instructions should not be able to steer the judge's score.

Test yourself

You are evaluating a new prompt for the customer support assistant. The LLM judge reports a 10% improvement in correctness, but human spot checks show no difference. What could be going wrong, and how would you investigate?

Answer: The judge may be biased toward the new prompt's style: longer, more confident replies often score higher even when the content has not improved. First, check whether the rubric penalizes padding or overconfidence; if not, add a criterion with a negative weight and rerun. Second, compute the judge's agreement with human labels on a calibration set, using correlation and Cohen's kappa. If agreement is low, the judge is not aligned; refine the rubric or add examples to the judge prompt. Third, look at the scores per criterion: the judge might be inflating fluency while correctness is flat. Fourth, check noise: rerun both prompts and count the cases where they disagree, since a small gain on a small set can be chance. Finally, run a blinded human evaluation on a larger sample. If people still see no improvement, the judge's signal is noise and the rubric needs redesign.

Next: Dynamic Prompts: Context Injected at Runtime. It shows how to assemble a prompt from live data on every call, and how to keep it cheap with prompt caching.

If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.

Sources

• Autorubric: A Unifying Framework for Rubric-Based LLM Evaluation on Non-Verifiable Tasks (Rao and Callison-Burch 2026)
• promptfoo (GitHub)
• DeepEval (GitHub)
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (Liu et al. 2023)
• Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreement (Han et al. 2025)
• OpenAI Evals (GitHub)
• AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents (Wang et al. 2025)