LLM-as-a-judge
Using a language model as the grader in an evaluation: it reads an output, applies a rubric or compares against a reference, and returns a score or verdict.
The only grading approach that scales to freeform output, and the one whose verdicts mean nothing until they have been checked against human labels. Judges have measured structural biases: they can favour whichever answer is presented first, prefer longer answers with no more content, and endorse wrong arithmetic. Mitigations are mechanical: run pairwise comparisons in both orders, supply the reference answer so the judge compares rather than reasons, score each rubric dimension separately, and allow an explicit cannot-determine verdict. A judge is a component under test, re-measured whenever the judge model changes.
Explained in depth
What an eval actually is, the three families of grader and what each costs, how far an LLM judge can be trusted with the measured numbers, and where evaluation sits in a delivery pipeline.
AI evals - building the suiteHow to build an evaluation suite for an LLM system - what a case is, where the dataset comes from, which grader fits which check, how RAG and agent evals differ from single-answer evals, and why offline and online evaluation are two halves of one loop.