Golden cases from last week's failures

The playground is a polite room. Production is the queue at 4pm, the missing field, the tool that 429s, the operator who pastes three tickets into one box. Your first golden set should come from that week, not from a synthetic wishlist.

Twenty to fifty real traces

Pull last week's failures. Twenty to fifty is enough to start. Early systems move in big steps, so a small, mean set still shows the move. Add a few successes so you know what good looks like on your path. Two people who know the job should reach the same pass or fail. If they would argue, the case is mush.

Sit with the traces. Write what went wrong in plain language. Do not start with labels like reasoning issue. Use labels a regression can assert: missing identity check, date argument mismatch, retried the same tool after a 429, stopped before the write.

Error analysis before more judges

Cluster those notes. Five or six named failure modes beat a dashboard of scores. Then decide whether you even need a model-as-judge. A lot of agent work has a deterministic check: did the row update, did the tool fire once, did the payload match the schema. Build that first. A second LLM grader on an unread pile just scores a story.

A judge you add before you have read the failures is scoring a story.

The fixture lives next to the feature

Promote one golden case per cluster. Input, expected state, the failure mode, a reviewer note, a pointer back to the source trace. Store it in the repo, beside the code that can regress it. A side Notion page dies the first week nobody opens it. A JSON file and a script you run in CI do not.

Do not make the golden the exact old transcript. Models paraphrase a correct answer. Store the invariants: the tool that must fire, the field that must be set, the refusal that must hold.

Agent evals are not single-turn tests

A one-turn chat is a different product from a path that calls tools, retries, and writes to a system of record. Score the path: which tools fired, whether the write happened, whether the agent looped after a transient error. Replaying the old assistant string as the expected output fails the first time the wording changes and the job still completed.

Ignore the leaderboard

Public model ranks never saw your identity model, your permission boundary, or your ugly Tuesday. A new name on a blog chart is not a reason to skip the fixture run. Taste still counts. A rude draft is a product defect. Put it in the set.

The loop

1. Read twenty to fifty traces from last week. Cluster the failures in language a reviewer would use.
2. Promote one case per cluster into a file next to the feature.
3. Prefer a code check over a judge. Add a judge only where two humans already agree.
4. Rerun the set when the prompt, the model, or a tool changes.
5. When a new failure shows up twice in production, promote it.

That is the eval work we will take under AI Development in Services: a set you can regress, grown from the traces you already have. Start your build if you want a pod that starts there instead of a platform migration.