MODULE 08 OF 12 · RELEASE 2 · OUTLINE

Evaluation Engineering

Proving a change made things better, instead of hoping it did.

This module is an outline. The plain-language explanation and the lab specification below are final. The full written lesson and its runnable lab repository are still being prepared, and enrolment is not open.

A chef who changes a recipe tastes both versions side by side. They do not change five things at once, serve it, and decide from the fact that nobody complained that tonight’s version is better.

So what is this really about?

AI systems are unusually easy to fool yourself about. You tweak the wording, try two examples, and they look better — so you ship. Then a whole category of input quietly breaks. The fix is a fixed set of examples, run every time, with the results compared against last time.

A change runs against the same evaluation cases each time; the build fails if a previously passing case regresses, even when the average improves.

QUICK CHECK

After a prompt change your average score rises from 82% to 86%, but three cases that used to pass now fail. Should you ship?

The words you will hear

Evaluation set: Your fixed tasting menu. The same examples every time, including the ones that used to fail.
Baseline: Last known score. New changes have to beat it, or at least not lose to it.
Regression: Something that used to work and now does not.
Model judge: Using an AI to grade answers where there is no exact right answer — after checking it agrees with humans.

By the end you will be able to

Build a fixed set of examples drawn from real failures.
Run it automatically and block changes that break things.
Use an AI grader only where you have checked it agrees with people.

How it is put together

An evaluation harness runs a fixed set of cases against the current system and reports per-case results and an aggregate. Deterministic properties are asserted exactly; subjective properties use a model judge with its own rubric and its own measured agreement with human labels. Results are stored per commit so a regression is attributable.

Where you start

A handful of manual test prompts run by hand before each release.

The lab

Convert manual prompts into a structured case file with expected properties.
Add cases drawn from real failures, not only from cases that already pass.
Assert deterministic properties exactly; reserve the model judge for the rest.
Measure the judge against human labels and record the agreement rate.
Run the harness in CI with a baseline that fails the build on regression.

What goes wrong, on purpose

The scenario. A prompt change improves the aggregate score while breaking a specific class of input.

What should happen. Per-case reporting surfaces the regression even though the aggregate improved, and the build fails on the affected cases.

How you prove it works

A test asserting the harness fails when a known-good case regresses.
A judge-agreement measurement recorded alongside the results.
A CI run producing a per-case report, not only a single number.

What you walk away with

An evaluation harness in CI, a baseline that blocks regressions, and a measured agreement rate for any model judge you rely on.