Evaluation engineering

Why this is the highest-leverage work

Deterministic software has a property LLM systems do not: you can tell whether it is correct. A function either returns the right value or it does not, and a test asserts it.

An LLM system's output is a distribution over acceptable answers, and "acceptable" is frequently a judgement. That has a consequence people underestimate until it bites: without a measurement, you cannot tell whether a change helped.

What happens without one is consistent across every team I have seen do it. Someone edits a prompt, reads three outputs, decides it seems better, and ships. A week later someone else improves something different and a behaviour that used to work no longer does. Nobody can say when it broke or which change did it, because there was never a number. The product changes constantly and improves randomly.

The argument for building it first is not rigour, it is speed. With an evaluation suite you can refactor prompts without fear, compare models in an afternoon, adopt a new base model when it ships, and answer "did that help?" with evidence. Every one of those is faster than the alternative, and the gap widens over the life of the product.

Building the golden set

Everything rests on this, so it is worth doing deliberately.

Use real inputs. From support tickets, query logs, user interviews, or the humans currently doing the work. Examples you invent are shaped by your assumptions about the task, which are exactly the assumptions you need the evaluation to test. A set of invented examples produces a system that handles invented examples.

Size: 50 is useful, 200 is comfortable, 1000 is a research budget. What matters far more than size is coverage, and adding a hundred more examples of the case you already handle adds nothing.

Stratify deliberately. Your set should contain, by design rather than by luck:

The common case, in proportion. The hard-but-valid cases where the answer requires real work. Ambiguous inputs where the correct behaviour is to ask a clarifying question. Inputs the system should refuse or say it cannot answer — a system that always answers has not been tested, and this is the slice that most often exposes the difference between a good system and a confident one. Adversarial inputs, including injection attempts. And known past failures, added as they are found, so a fixed bug cannot return unnoticed.

Define what good means, precisely enough for someone else to apply it. "A good summary" is not a specification. "Includes the decision and the owner, under 100 words, no information absent from the source" is. If two people on your team would score the same output differently, the rubric is not finished.

Version it and freeze it. The evaluation set is a measuring instrument. Changing it changes what the numbers mean, so changes should be deliberate, recorded, and accompanied by a re-baseline rather than slipped in.

Decompose before you measure

A single end-to-end quality score is the most common evaluation design and the least useful one, because when it moves you do not know why.

Measure each stage that can fail independently:

# A RAG pipeline has at least four measurable stages.

router      → did it correctly decide whether to retrieve?
retrieval   → recall@k, precision@k, nDCG@k
generation  → faithfulness to the retrieved context
end-to-end  → did the user get a correct, useful answer?

# The ordering matters: a failure upstream makes every
# downstream measurement meaningless. There is no point
# debugging generation while retrieval is broken.

The payoff is diagnostic. When end-to-end quality drops, the stage metrics tell you immediately which component regressed, and you skip the day of bisecting by hand. It also tells you where the ceiling is: if recall@10 is 0.6, generation quality cannot exceed what 60% coverage allows, and improving the prompt is wasted effort.

The same applies to agents — per-step success and end-to-end success are different numbers, and the gap between them is where error compounding lives. A 95% per-step rate over ten steps is roughly 60% end to end, and only measuring both makes that visible.

Programmatic checks first

Before anything judges anything, write the assertions a computer can make. They are free, deterministic, instant, and they catch a surprising share of real defects.

# These need no model and no human. Run them on every output.

def check(output, context, question):
    assert is_valid_json(output)                     # schema
    assert set(output) >= REQUIRED_FIELDS
    assert len(output["summary"].split()) <= 100     # length
    assert output["citations"]                        # cited at all
    for cid in output["citations"]:
        assert cid in context.chunk_ids                # not fabricated
    assert not PII_PATTERN.search(output["summary"])
    assert detect_language(output) == expected_language

The citation check deserves particular attention, because fabricated citations are one of the most damaging failures a grounded system can have — the answer looks verifiable, which makes users trust it more, not less. Validating that every cited id exists in the retrieved set is three lines and it makes that failure impossible to ship.

Beyond assertions, the deterministic metrics: exact match where there is one right answer, numeric tolerance for extracted figures, set overlap for extracted entities, and reference-based text metrics where a reference answer exists — with the caveat that string-overlap metrics correlate poorly with quality on open-ended generation and should not carry much weight.

Every check you can express in code is one you never pay a model to perform, never wait on, and never argue about.

LLM-as-judge, calibrated

For the genuinely subjective remainder — is this summary faithful, is this tone right, is this answer helpful — a model scoring outputs is the practical option at any scale.

It is also a measuring instrument, and an uncalibrated instrument produces confident numbers that mean nothing. The known biases are specific and worth designing around:

Position bias. In pairwise comparison, judges favour one position. Evaluate both orderings and average; a large disagreement between orderings is itself a signal that the two outputs are close.

Verbosity bias. Judges prefer longer answers largely independently of quality. Either control for length in the comparison or instruct the judge to disregard it — and then verify the instruction worked by checking whether score correlates with length.

Self-preference. Judges score outputs from their own model family more highly. Using a different model to judge than to generate is cheap insurance against measuring your own reflection.

Scale compression. Asked for 1–10, judges cluster in 6–8 and the scale carries less information than it appears to. Binary or three-point rubrics with explicit criteria are substantially more reliable.

Writing a judge prompt that works

Give it the criterion, not the question. "Rate this answer's quality" invites the model's general aesthetic preferences. "Does every factual claim in this answer appear in the provided context? Answer yes or no, then list any claim that does not" is a checkable question with a defensible answer.

Ask for the reasoning before the verdict — a judge that commits to a score and then explains it rationalises, while one that examines first and concludes after is more accurate. Provide the rubric with a worked example of each category. And request a structured output so parsing is not its own failure mode.

Is the difference real?

Your new prompt scores 0.84 against the old one's 0.81 on 100 examples. Ship it?

Not on that alone. On 100 examples, a three-point difference is comfortably inside the noise, and shipping it is a coin flip you have dressed up as a decision.

The right tool is a paired bootstrap, and the important word is paired: both variants are evaluated on the same examples, so their errors are correlated and comparing them directly has far more statistical power than comparing two independent intervals.

# Paired bootstrap over the per-example score difference.
def bootstrap_diff(scores_a, scores_b, n=10000):
    diffs = np.array(scores_a) - np.array(scores_b)   # paired
    means = [np.mean(np.random.choice(diffs, len(diffs), replace=True))
             for _ in range(n)]
    return np.percentile(means, [2.5, 97.5])

# If the interval straddles zero, you have not established
# a difference. That is a result, not a failure — it tells you
# the change is not worth the risk, or the set is too small.

Two further cautions.

Multiple comparisons. If you tried forty prompt variants and are reporting the best, the winner is optimistically biased by selection regardless of what any single test says. Confirm on a held-out set that took no part in the search.

Power. Before running the comparison, ask what effect size your set can detect. If a three-point difference is undetectable at n=100, then either enlarge the set or accept that you are choosing between variants you cannot distinguish — and make that choice on cost or latency instead, which at least are measurable.

Wiring it into CI

An evaluation suite that runs when someone remembers is not a safety net. The value comes from it being unavoidable.

Every commit — the fast, deterministic layer: programmatic checks, schema validation, and a small prompt regression suite with fixed inputs. Seconds to a couple of minutes. No model calls if you can avoid them, or cached ones if you cannot.

Every pull request touching prompts, retrieval or model config — retrieval metrics and the cheaper evaluation set, with results posted to the PR so the reviewer sees the quality delta next to the diff. A prompt change that drops recall by four points should be as visible as a failing unit test.

Nightly and on release candidates — the full suite including judge-based scoring, plus segment breakdowns. Expensive, slow, and it does not need to block a commit.

Two things make this work in practice and their absence is why most attempts die:

Determinism. Pin the model version, pin dependencies, fix seeds, set temperature to zero for evaluation, and cache what you can. A flaky quality gate gets disabled within two weeks and then you have nothing — this is the single most common way evaluation infrastructure is lost.

Thresholds with hysteresis. Fail on a meaningful regression, not on any movement. Set the gate outside the noise band you measured, so it fires on real changes and not on resampling.

Online evaluation

Offline evaluation is a fixed sample of a shifting reality. It will not catch everything, and the gap between it and production is where most unpleasant surprises live.

Explicit feedback — thumbs up and down. Low volume, heavily biased toward extremes, and still useful because the negatives are a direct feed into your golden set.

Implicit signals, which are higher volume and often more informative. Did the user click a citation. Did they copy the output. Did they immediately rephrase and ask again — a rephrase directly after an answer is one of the strongest available signals that the answer was bad, and it costs nothing to instrument. Did they escalate to a human. Did they complete the task.

Production sampling. Take a random sample of real traffic daily, run your judge over it, and track the score. This catches drift that a fixed evaluation set structurally cannot, because the set does not change and your traffic does.

A/B tests for changes significant enough to warrant one, with the business outcome as the primary metric and the model metric as a diagnostic.

Then close the loop, which is the part that turns evaluation from a gate into an engine: every production failure you investigate becomes a new example in the golden set. That is what makes the suite get better at catching the failures you actually have, rather than the ones you imagined at the start.

Anti-patterns

Vibes-based iteration. Reading a handful of outputs and forming an impression. Fine for exploration, disastrous as a decision procedure, and it is the default state of most teams.

Evaluating only the happy path. No refusals, no ambiguity, no adversarial inputs. Produces a system that is excellent until it meets a real user.

Optimising the judge instead of the system. If your judge is the target, you will eventually find the prompt that pleases it rather than the one that helps users. Calibration against human labels is what keeps the judge anchored to something real.

One number for everything. Faithfulness, helpfulness, format compliance and safety fail independently and need separate measurement. A single blended score lets a regression in one hide behind an improvement in another.

Letting the evaluation set drift into the training set. If you iterate against a set long enough you are fitting it, and its number stops predicting production. Keep a held-out set you look at rarely and deliberately.

Measuring what is easy instead of what matters. Token overlap with a reference is easy and correlates weakly with usefulness. It is better to have a coarse measure of the right thing than a precise measure of the wrong one.