AI evals - building the suite

Ordinary software is tested by asserting that a known input produces a known output. An LLM system breaks both halves of that sentence. The same input produces different outputs on different runs, and for most useful tasks there is no single correct output to compare against - a good support reply, a good summary, a good plan can each be written several ways. The system also fails in ways that never raise an error: it states something false with full confidence, follows an instruction it should have refused, or calls the right tools in the wrong order and still returns a plausible answer.

An eval is the test that survives those conditions. It fixes an input, runs the system, and applies grading logic to what came out - where the grading logic is allowed to be a rule, a rubric, a reference answer, a second model, or a person. This page is about building that suite: what goes in the dataset, which grader fits which check, how the shape changes for retrieval systems and agents, and how the suite keeps learning after launch.

The companion page covers the question this one leaves open: how far a model grader can be trusted, and what the measured evidence says about its biases. Read that before you let a judge gate a release. Read this first.

Model evals and system evals are different activities

The word “eval” covers two things that share tooling and almost nothing else.

Model evals measure a foundation model’s raw capability on shared public tasks - reasoning benchmarks, coding suites, factual recall sets. They exist so that models can be compared with each other, and they are the right thing to look at once, when choosing a base model. A model that leads a leaderboard is a reasonable starting point.

System evals measure whether your application does its job on your inputs. The system is the model plus everything wrapped around it: the prompts, the retrieval layer, the tool definitions, the routing logic, the post-processing. A benchmark score tells you nothing about that assembly, because the benchmark never saw your prompts or your data. A model that leads the leaderboard can still fail your cases, and a smaller model with a better retrieval layer can pass them.

Everything below is about system evals. The practical test for whether you are doing one: if the inputs came from your product rather than a public dataset, it is a system eval.

The anatomy of a case

A suite is a collection of cases. Each case has a fixed shape, and getting the shape right early saves a rewrite later.

The field that is most often missing is the second one. Teams collect inputs readily, because inputs are lying around in logs, and then discover at grading time that nobody wrote down what the right answer was. A case without an expected behaviour is not a test; it is a sample. It can still be useful for reference-free scoring (below), but it cannot tell you whether a change was an improvement.

The field that is most often ignored is the trace. For a single-turn question-answering system the final answer is most of what matters. For anything that retrieves or acts, the final answer is the least informative part of the run, and a suite that grades only the answer will pass an agent that reached the right result through an action it should never have taken.

Where the dataset comes from

There are four sources, and a mature suite draws on all of them.

Written by hand from the requirements. At the start of a project the product requirements convert directly into cases: this is what the feature must do, here is an input that exercises it, here is what good looks like. This is the cheapest moment to write cases and the one with the highest yield, because the intent is still explicit. Later, cases have to be reverse-engineered from a system already in front of users.

Curated from historical data. If the task was previously done by people - support agents answering tickets, analysts writing summaries - their outputs are reference answers that already exist. The work is selection and cleaning, not authoring. The closer these are to real traffic, the more the suite’s score means.

Promoted from production failures. Once the system is live, every failure a user reports and every trace that scores badly is a candidate case. This is the source that keeps a suite honest, because it tests the failures nobody anticipated rather than the ones the team imagined. A failure that reaches a user and does not become a case will happen again.

Generated synthetically. A model can read your documents and write plausible questions about them, which is useful for volume and for coverage of corners no human would think to ask about. Treat the output as a draft. Synthetic cases tend to be well-formed, on-topic and polite, which is precisely what real user input is not, and a suite built only from them measures how the system handles a user who does not exist.

Two properties matter more than the source. The set has to be versioned, so that a score from this week can be compared with one from last week; a dataset that changes silently makes every trend meaningless. And it has to be held partly out. A team that iterates against the whole set will tune the system to the set, and the score will climb while the product does not improve. Keep a slice the team does not look at.

Two ways to grade: with a reference and without

Every grader is one of two kinds, and the choice is forced by what you have.

Reference-based grading compares the output to a known-good answer. This is the natural mode when a correct answer exists - a classification label, an extracted field, a computed figure, a specific document that should have been retrieved. The comparison can be exact, or tolerant of wording, or made by a model asked “do these two answers say the same thing”. Reference-based grading is what makes regression testing possible: you can say the system got this right before and gets it wrong now.

Reference-free grading scores the output directly against criteria, with no expected answer in hand. Is it grounded in the retrieved context? Does it answer the question that was asked? Is it in the right register? Does it contain anything the policy forbids? This is the only mode available in production, where users supply the inputs and nobody has pre-written the answers, and it is the mode that fits open-ended tasks even offline.

The two are not competitors. A suite runs reference-based checks on the cases that have references and reference-free checks everywhere, and the production monitor runs reference-free checks alone. The mistake is to build only the first kind and then have nothing that runs after launch.

Choosing the grader

Graders come in three families, and the practical question is which one carries the verdict for a given check. The answer depends on what is being checked, not on preference.

Code graders are deterministic, free to run, and certain when they fire. Anything that can be expressed as a rule belongs here: the output parses as JSON, the required keys are present, the answer is under the length limit, the classification matches the label, the cited chunk is one of the retrieved chunks, the tool called is on the permitted list. Run them first, on every case, because they catch a large share of real regressions for nothing and their verdict needs no calibration.

Model graders cover what code cannot express. Whether an answer is helpful, whether it is fully supported by the context, whether a summary preserves the meaning of the source, whether a refusal was appropriate - these are judgements, and a strong model prompted with a clear rubric is the only grader that scales to them. They cost money on every case on every run, they are not deterministic, and their verdicts mean nothing until they have been checked against human labels on your own data. The companion page sets out what that check has found.

Human graders are the standard everything else is calibrated against, and they do not scale. Their role in a running suite is not to grade every case but to grade a labelled sample - a few dozen to a few hundred - that the automated graders are periodically compared with. When the model grader and the human sample disagree more than they used to, the grader has drifted, and its scores stop being trustworthy until the disagreement is understood.

Three habits make the model grader more useful. Score dimensions separately rather than asking for a single number, because a single number cannot say which dimension moved. Supply the reference answer when one exists and ask the grader to compare rather than to reason, since comparison is what it does well. And give it an honest way out - an explicit “cannot determine” - because a grader denied one will manufacture a verdict.

What changes for retrieval systems

A retrieval-augmented system has two stages that fail independently, and a suite that scores only the final answer cannot tell which one failed.

The first row is the one to build first. Retrieval failures are the most common cause of a wrong answer in these systems, they are cheap to detect with a code grader once the relevant chunks are labelled, and a fix at the retrieval stage is usually a configuration change rather than a prompt rewrite. A generation grader run on a retrieval failure produces a misleading result: the model was grounded in the wrong context, answered faithfully from it, and gets scored as if the fault were its own.

The third row - grounding, sometimes called faithfulness - is the check that catches the failure people most associate with these systems: a fluent answer that the retrieved evidence does not support. It is a reference-free check, which means it runs in production, and it is the single most useful online monitor a retrieval system can have.

What changes for agents

An agent does not produce an answer; it produces a trajectory - a sequence of model proposals, tool calls, tool results and eventually a final state. Grading the final state alone misses most of what can go wrong.

The final answer can be right for the wrong reasons. An agent asked to fix a failing test can make the test pass by deleting it. The output is green. The trajectory shows the deletion. Only a grader with access to the trace can fail this case.

The right tools in the wrong order is a failure. A booking agent that charges the card before confirming availability has done both steps correctly and the task wrongly. Trajectory graders check ordering and preconditions, not only presence.

Unauthorised actions are failures regardless of outcome. A tool call outside the permitted set, a write where only reads were allowed, an escalation that skipped the approval gate - these fail the case even when the task was completed. This is a code grader: the permitted set is a list, the calls are a list, and the comparison is exact.

Not finishing is a valid outcome. An agent that stops and reports that it cannot proceed safely has done the right thing in many scenarios, and a suite that scores only task completion penalises the behaviour it should reward. Cases need to say what the right stopping point is, not only what the right answer is.

For agents that write and run code, the eval has to run the code. Grading the agent’s description of what it did is grading its prose. The environment - a sandbox with the real filesystem, the real test runner - produces the evidence: exit status, files changed, tests passed. Deterministic checks on that evidence come first; a model grader on whether the plan was reasonable comes after.

The final property that changes: variance is larger. A single-turn system is somewhat non-deterministic. An agent compounds that over every step, so the same case can succeed on one trial and fail on the next with no change to anything. Several trials per case is not optional here, and the score to report is the distribution, not the mean.

Offline and online: one loop

Evaluation is usually described as two activities - testing before release and monitoring after - and the description is accurate about the mechanics while missing what connects them.

Offline, you own the inputs. The dataset is curated, the references exist, several trials per case are affordable, and the question is comparative: is this version better than the last one on the things we have decided to measure. The output is a gate - a threshold against a baseline that a candidate must clear before it ships.

Online, users own the inputs. There are no references, the volume is high, and every trace is real. Grading is reference-free and sampled: a share of production traces scored for grounding, policy compliance, refusal correctness, or whatever the named failure modes are. The question is not comparative but absolute: is what users are getting still acceptable, and has it changed.

The arrow that closes the loop runs from online failures back into the offline dataset. A trace that scored badly, a complaint, a cluster of similar failures - each is examined, named as a failure mode in product language, labelled, and added as a case. The next offline run then tests for it, and the fix that addresses it is gated on it. Without that arrow the two activities are separate systems that age at different rates: the offline suite tests last year’s anticipated failures while production produces this year’s real ones.

Building the suite from one failure

Nothing above requires a complete suite to begin. The unit of progress is a single named failure, and the loop can be run once on one trace before it is run continuously on thousands.

The second stage carries the weight. “Helpfulness”, “accuracy” and “coherence” feel like metrics until two people score the same trace differently and the team discovers the word never had a shared definition. A failure mode named specifically enough that reviewers agree on it is the precondition for everything downstream: the labels, the grader, the monitor, the gate.

Failure modes of the suite itself

• Cases with no expected behaviour. Inputs were collected, references were not written, and at grading time the only option is reference-free scoring of everything. Detect it by counting cases with an empty expected field. Fix it by writing references for the subset that has a correct answer, starting with the cases that came from incidents.
• The dataset is all synthetic. Every input is well-formed and on-topic, so the suite measures a user who does not exist. Detect it by sampling real traffic and checking how many cases resemble it. Fix it by promoting real traces.
• Only the final answer is graded. Retrieval failures are scored as generation failures, and agent trajectories with unauthorised actions pass. Detect it by asking whether any grader reads the trace. Fix it by adding the stage-level checks in the tables above.
• The judge was never measured. A model grader’s scores are treated as ground truth from the day it was written. Detect it by asking when its agreement with human labels was last checked. Fix it with a labelled sample and a periodic comparison, re-run whenever the judge model changes.
• One trial per case. The score moves between runs with no code change, the team learns the suite is noise, and stops reading it. Detect it by re-running the same version twice. Fix it with several trials and a reported spread.
• Nothing runs online. The suite is reference-based, so it cannot run on production traffic, and the first sign of drift is a complaint. Detect it by asking what scores live traces. Fix it with reference-free graders for the named failure modes.
• The loop is open. Production failures are fixed but never become cases, so the same failure returns after the next model upgrade. Detect it by checking whether the last three incidents have corresponding dataset entries. Fix it by making the case part of the definition of done for the fix.

Trade-offs

The suite costs money to run, and model grading is most of the bill. Every model-graded case is an inference call, on every run. The proportionate response is to push everything expressible into code graders and reserve the model for judgements, to sample rather than grade all production traffic, and to keep a small fast subset for per-commit runs with the full suite reserved for release qualification.

Defining “good” is a product decision, and the suite forces it early. Two competent engineers will disagree about whether an open-ended answer was acceptable. That disagreement has to be resolved into a rubric before a grader can be written, and the resolution is not a technical matter. Evals do not remove the argument; they move it from after launch to before, and put it in writing.

Coverage and speed pull against each other. A suite large enough to be statistically meaningful is too slow to run on every change; a suite fast enough for every change is too small to detect a subtle regression. The resolution is layering - a small suite on every commit, a full suite at the gate, sampled monitoring in production - rather than a single suite that tries to be both.

A measured system is not a safe one. Quality evaluation asks how the system behaves under expected use. It says nothing about what a motivated adversary can make it do. Red teaming is the other activity, with its own methods, and neither substitutes for the other.

What this is not

Not a benchmark score. Public benchmarks compare models on shared tasks and are the right thing to consult when choosing a model. They do not run on your inputs, do not see your prompts, and cannot tell you whether your system works.

Not observability, though it depends on it. Tracing records what the system did; evaluation says whether the result was any good. A team with complete traces and no graders can reconstruct exactly how a bad answer was produced without ever having established that it was bad. Build the tracing first, because the graders need it, and then build the graders.

Not a one-time gate. A suite written at launch and never extended tests the failures anticipated at launch. Its value comes from the loop - from production failures becoming cases - and a suite that is not fed decays into a check that always passes.