The number worth knowing
Anthropic's Sep 28 post on automating eval design walks through tuning one of their own skills against an eval. The score went from 66.1% to 87.9% in three phases of work. The last phase was fixing graders, plus more skill edits.
Two of the broken pieces, as they describe them: one task asked for code that catches one error type, while its grader wanted a chain of at least three. Another grader's instructions contradicted their own docs, and testing the real API showed the docs were right.
How to spot a broken grader (or case)
- A case that fails every run, no matter how many times you repeat it. Anthropic calls that a common tell of an impossible or ambiguous task.
- A case that never improves, even after you fix the obvious gaps. That one is a tell that the case or the grader is flawed.
- Verdicts that flip. Their eval builder runs the grader twice on the same output and reports whether the verdict changed.
- Infrastructure noise. Timeouts, API errors and cut-off answers can pass as model variance if nobody checks the plumbing.
Anthropic's January guide to agent evals says the same thing from the other side: you won't know if your graders are working unless you read the transcripts and grades from many trials.
Keep some cases out of the loop
Tuning against an eval is a loop: read the failures, make one change, run it again, keep it or revert it. The danger is fitting the cases you can see. The fix is a split: a train set the loop may read, and a test set it never sees.
- Each round makes one change, aimed at the most common failure.
- Train and test both go up: keep it. Train up, test flat: that's overfitting, so revert.
- Score stalls for two or three rounds: read every remaining failure and sort it by cause first.
- Finish on the version that did best on the test set, not the train set.
Their cost example shows why it matters. A support benchmark had 44 tickets: 30 for the search, 14 held out. On the 14 it never saw, the final setup scored 90.5% vs 78.6% for the original, at about a fifth of the cost.
Break your code on purpose
Graders aren't the only checks that can check nothing. Your tests can too. Mutation testing finds out: make small deliberate bugs in your code (mutants) and see whether a test goes red. Google wrote up running it inside code review back in 2018, for 6,000 engineers on the changes they author or review.
I hit both problems on a personal project. The crash detector in my mutation runner was dead code, so a crash could get counted as a caught bug (fixed since: a crash never counts). And 15 of 34 hand-made subtle mutants slipped past my first tests. Each one now has a test that catches it.
Here's what a subtle mutant looks like. This is a made-up example, not from that project:
# the code if retries > max_retries: stop() # the mutant: one operator changed if retries >= max_retries: stop() # a test that only checks "it stops eventually" passes both. # the test that catches it checks the edge: assert not stopped_after(retries=max_retries)
The checklist to copy
- Run the grader twice. If it disagrees with itself, fix the grader first.
- A crash is a fail, never a pass.
- Keep some test cases out of the tuning loop.
- Break your code on purpose (mutation testing) and see if a test goes red.
- Hand-write a few sneaky cases, not just obvious ones.
A score is only as honest as the grader behind it. What's the last check you found that was checking nothing? Tell me on LinkedIn.
Sources and further reading
Every number on this page comes from these. The personal-project details are my own.