Evaluating LLM systems

An evaluation is a test for an AI system: give it an input, then apply grading logic to the output to measure whether it succeeded (Anthropic, 2026). Hold onto that definition, because it is unremarkable. It is the same shape as any other automated test.

What makes evaluation its own subject is the grading logic. An ordinary unit test compares output to an expected value, and that comparison is free and certain. Here the expected value is often a range of acceptable answers, the comparison is a judgement, and the thing making the judgement may be another model with failure modes of its own. Everything difficult about evals follows from that single substitution: the oracle stopped being free and became a component.

So an eval suite has two jobs, and teams routinely do only the first. The first is measuring the system. The second is establishing that the measurement is worth anything.

The shape of an eval run

One eval is a task, one or more graders, and a trial that produces a transcript for those graders to read. The vocabulary is worth being precise about, because the transcript is what separates this from ordinary testing. It is the complete record of a run, including tool calls, reasoning and intermediate results, and not only the final answer.

Steps four and five carry most of the practical weight. The rest of this page is largely about them.

Three families of grader

Graders divide into code-based, model-based and human, and the division matters because they fail differently and cost differently. Anthropic’s account sets them out with the trade-offs attached, and the summary below follows that division.

The practical ordering follows from the table. Deterministic checks first, because a citation that does not resolve to a retrieved chunk, a JSON payload that does not parse, or a required field that is absent are all cheap to detect and unambiguous when detected. They catch a large share of real regressions for nearly nothing. Model grading is for what is left, which is usually the part that actually matters: whether the answer was any good.

LLM-as-judge, and how far to trust it

Using a strong model to grade output is the only approach that scales to freeform answers, and the question is not whether to use it but how much weight its verdicts can carry. There is measured evidence on this, and it is more encouraging and more alarming than most summaries suggest.

The encouraging part: on open-ended questions, a GPT-4 judge agreed with human preferences at over 80%, which the authors note is the same level of agreement humans reach with each other (Zheng et al., 2023). Their figures come from 3,000 controlled expert votes and 3,000 crowdsourced votes. If human-to-human agreement is the ceiling, a judge at that level is not a poor substitute for human review; it is approximately as consistent as the thing it replaces.

The alarming part is what the same paper found when it attacked the judges.

Position bias. Present two answers, then swap their order and ask again. A judge free of position bias returns the same verdict both times. Consistency measured as follows:

Read the first row again. With the default prompt, Claude-v1 gave a consistent answer under a swap in under a quarter of cases, and favoured whatever was presented first three times out of four. Even GPT-4 with the renamed prompt, the strongest configuration tested, left a third of its verdicts changing on presentation order alone. The authors suspect the cause lies in training data or in the left-to-right architecture itself, and leave it open.

The mitigation is mechanical rather than clever: run every pairwise comparison in both orders and count only the cases where the verdict holds. That doubles the grading cost and converts a silent bias into a visible disagreement rate, which is a good trade.

Verbosity bias. The authors built a “repetitive list” attack: take an answer containing a numbered list, then pad it so a five-item response becomes ten items where the extra five are rephrasings of the original five. No new information, more words. The attack counts as successful if the judge prefers the padded version.

Two of the three judges were fooled in over nine cases out of ten. That is worth sitting with, because a judge with this bias does not merely score badly. It actively rewards padding, so a team optimising against it will ship a system that says more and means the same.

Limited reasoning, and the fix that worked. On arithmetic questions, judges endorsed answers that were simply wrong. Adding chain-of-thought instructions helped; supplying a reference answer helped considerably more.

That last row is the most useful finding on this page. When the correct answer can be computed, give it to the grader instead of asking the grader to work it out. The judge’s job becomes comparison, which it is good at, rather than reasoning, which it is not.

What these numbers do and do not license

The models tested were Claude-v1, GPT-3.5 and GPT-4, and this was 2023. Current models are stronger, and the specific percentages should be assumed stale. So state plainly what carries forward and what does not.

The mechanisms carry forward: a judge with an ordering preference, a length preference, and weak arithmetic is a judge whose failures are structural, and nothing about a larger model guarantees those are gone. The magnitudes do not carry forward, and quoting 91.3% as though it describes today’s systems would be exactly the error this site exists to avoid.

The consequence for practice is the same either way, and it is the point of the whole section: a judge is a component under test. Measure its agreement with human labels on your own data before you let it gate anything, and re-measure when you change the judge model. Anthropic’s guidance says the same in operational terms, holding that a model grader has to be kept in close calibration against expert human judgement so that the two can be trusted not to drift apart.

Two smaller techniques worth taking from the same guidance. Leave the grader a legitimate escape, a permitted “Unknown” for cases where the evidence does not reach a decision, because a grader denied one will manufacture a verdict instead. And write structured rubrics that score each dimension separately, rather than asking for one overall number, since a single score cannot say which dimension moved.

Where evaluation sits in delivery

Evals are usually discussed as a development activity, which understates them. The same suite does different work at different points, and the two positions have different requirements.

The last stage is the one most often missing, and its absence is why suites decay. A curated set written at launch tests the failures you anticipated then. Only production failures tell you about the ones you did not, and Anthropic’s observation is that the job only gets harder with delay: at the start the product requirements convert straight into cases, whereas later you are inferring backwards, from a system already in front of users, what it was supposed to do in the first place.

Sample size, and what a number can support

Two trials of one case tells you nothing you can defend. This is not pedantry about statistics but the practical reason eval results get ignored: a suite that moves by five points on a rerun with no code change teaches people that the suite is noise.

The useful framing is effect size. Early in development, changes tend to be crude and their effects large, so a small set detects them and there is no reason to build more. As a system matures the remaining changes are subtler, the effects smaller, and a set that was adequate becomes unable to distinguish improvement from variance. Anthropic’s guidance approaches it from the other side, suggesting you begin from whatever is already being checked by hand, accept that a rough majority of the value lands first, and grow the set once it stops telling you anything new.

Two consequences follow, and both are cheap to implement:

• Run each case several times and record the spread, not just the mean. The spread is what tells you whether a difference between two runs is a result.
• Report the continuum. A binary pass rate discards the difference between failing at the first step and almost succeeding, which is precisely the signal that shows whether work is heading in the right direction.

Failure modes

• The judge was never validated. A model grader is deployed and its scores are treated as ground truth from day one. Detect it by asking when its agreement with human labels was last measured; if the answer is never, the suite is measuring the judge’s preferences. The fix is a human-labelled subset and a periodic agreement check, re-run whenever the judge model changes.
• Pairwise comparisons run in one order. Position bias then contributes silently to every result. Detect it by re-running a sample with the order swapped and looking at how often the verdict flips.
• Nothing is pinned. An eval run against “the latest model” cannot be reproduced, so a regression cannot be attributed to the change that caused it. Pin the snapshot and record it with the result.
• Optimising against the metric. The suite becomes the target and the system learns the suite’s preferences, including the ones nobody intended, such as length. Held-out cases the team does not iterate against are the only real defence, and static evals can be gamed in ways their authors did not anticipate: Anthropic reports a model solving a benchmark task by exploiting a loophole in the policy rather than by doing the work.
• Grading only the final answer. For anything agentic this misses the failures that matter, because a correct answer reached through an unauthorised action is a failure the output cannot show. Assert on the transcript, including which tools were called.
• A gate too flaky to keep. Variance exceeds the threshold, the build fails at random, and the gate is disabled. Everything after that is unprotected while appearing to be covered. More trials per case, or a wider threshold honestly chosen, both beat a gate that gets switched off.

Trade-offs

Evaluation buys the ability to make a claim about a change, and it is not free in three respects worth naming before committing.

It costs real money per run. Model grading is inference, charged per token, on every case, on every run, and a suite run on every commit is a recurring cost that scales with how thoroughly you grade. Deterministic checks are how you keep that bill proportionate: use them for everything they can express, and reserve model grading for what genuinely needs judgement.

Defining success is the hard part, and it is not a technical problem. For anything open-ended, two competent engineers reading the same output will disagree about whether it was good. That disagreement has to be resolved into a rubric before grading means anything, and the resolution is a product decision. Evals do not remove that argument; they force it to happen early and in writing, which is most of their value.

A measured system is not a safe one. Quality evaluation and adversarial testing answer different questions. A suite of representative cases says what happens under expected use; it says nothing about what a motivated attacker can extract. Red teaming is the other activity, and neither substitutes for the other.

What this is not

Not a benchmark score. Public benchmarks compare models on shared tasks, which is useful when choosing one and close to useless for telling you whether your system works on your data. A model that leads a leaderboard can still fail your cases, and the only thing that answers that question is a set built from your own inputs.

Not observability, either, though the two are often bought together. Tracing tells you what your system did and how long it took. Evaluation tells you whether the result was any good. A team with excellent traces and no graders can reconstruct exactly how it produced a bad answer, without ever establishing that the answer was bad.

And not a substitute for the governance work, though it produces its raw material. Frameworks that ask for evidence about a system’s behaviour, including the MEASURE function of the NIST AI RMF, are asking for what an eval suite generates. The evidence is a by-product of measuring for your own reasons, which is the right order to do it in.