Evals
Structured tests for an LLM system: a fixed input, a run, and grading logic applied to the output, where the grader may be a rule, a rubric, a reference answer, a second model or a person.
Ordinary tests assert that a known input produces a known output. An LLM system produces different outputs on different runs and, for most useful tasks, has no single correct output to compare against, so the assertion has to become a grader. Evals split into model evals, which compare foundation models on public benchmarks, and system evals, which measure whether a particular application does its job on its own inputs. A suite is a versioned dataset of cases plus the graders that score them, run offline against a candidate version and online against sampled production traffic, with production failures promoted back into the dataset.
Explained in depth
How to build an evaluation suite for an LLM system - what a case is, where the dataset comes from, which grader fits which check, how RAG and agent evals differ from single-answer evals, and why offline and online evaluation are two halves of one loop.
Evaluating LLM systemsWhat an eval actually is, the three families of grader and what each costs, how far an LLM judge can be trusted with the measured numbers, and where evaluation sits in a delivery pipeline.
Implemented by
- LangSmith
- Langfuse