Nine good-looking steps, one wrong ending
The ThinkingBox write-up opens with a late-delivery case: a $745 appliance delivery stuck in a carrier exception. The agent makes nine correct-looking tool calls, then closes the ticket as resolved. But the carrier exception was still open, so the right state was "on hold." Read the transcript and it looks fine. Read the database and it's wrong.
"A trajectory is a claim. Database state is the evidence."ThinkingBox, Microsoft, on the Hugging Face blog (Oct 3, 2026)
How ThinkingBox grades
- 507 stateful business workflows across retail, auto insurance, travel, a neobank and consulting.
- Every workflow runs 20 times per model, each attempt in its own isolated session with a simulated user.
- Deterministic judges check the final backend state and side effects. The evaluator stays separate from the agent.
Same idea as Anthropic's January guide to agent evals: the outcome is the final state of the environment, and it's often better to grade what the agent produced, not the path it took.
Three numbers, three different questions
| Metric | The question it answers | Kimi-K3 |
|---|---|---|
| pass@1 | How often does one run work, on average? | 57.37% |
| pass@20 | Did it work at least once in 20 tries? | 93.89% (476 of 507) |
| 20 / 20 | Does it work every single time? | 13.41% (68 of 507) |
Breadth and consistency pull apart. Kimi-K3 has the widest coverage and among the lowest consistency. Opus 5 is more consistent but solves fewer tasks at least once. And even the best overall pass@1 in their table, Claude Opus 5.5, is 67.16%.
The idea isn't new. The 2024 τ-bench paper proposed pass^k to measure reliability over repeated trials, and found gpt-4o function-calling agents were quite inconsistent: pass^8 under 25% in retail. ThinkingBox's 20/20 asks the same question, harder.
Clean exits are the dangerous ones
A clean exit with wrong field values, extra effects or missing effects is the worst kind of failure, because nothing flags it. ThinkingBox's read on all failed runs: about four in five were tool handling, not reasoning (each share is an average of per-model shares):
| Failure signature | Share |
|---|---|
| Tool usage (errors, failed preconditions, empty lookups) | 79.9% |
| Wrong state updates | 10.3% |
| Incomplete user resolutions | 7.0% |
| No state-changing action | 2.9% |
Same rule, my setup
I use the same rule on my personal AI setup. When I ran its acceptance tests, the score was 6 pass, 1 partial, 3 not verified, 0 fail. "Not verified" means it may have worked, but it left nothing on disk to check at check time. No trace, no pass.
One example: a test sent the setup a short note by text and expected a new file plus a log line. When I checked, nothing on disk had changed, so it went in the not-verified column instead of the pass column.
How to put it to work
- Grade the end state, not the transcript.
- Run it many times. One success is luck, 20 out of 20 is reliability.
- Treat a clean exit as unproven until the data agrees.
- Give "not verified" its own column.
- From ThinkingBox's own list: check the end state before committing an action, sort tool errors so retries go after the recoverable ones, keep the tool list to what the workflow needs, and get human approval for changes that are hard to undo.
"Done" is a claim. The data is the proof. How do you prove an agent actually did what it says it did? Tell me on LinkedIn.
Sources and further reading
Every number on this page comes from these. The acceptance-test score is my own.