A trial is a task plus its test suite. A model passes only when every test goes green — retries priced in, wall clock running. Each trial is a public page you can share. Tasks run easy → hard — watch the strikeouts grow.
| task | haiku-4-5 | gpt-5-mini | gpt-4.1 | sonnet-4-6 | gpt-5 | opus-4-8 | fable-5 | gpt-5.1 | gpt-5.2 | gpt-5.4 | gpt-5.5 | sonnet-5 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| T01 · Fix the failing test | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | R | ✓ | ✓ | ✓ | ✓ | ✓ |
| T02 · Access-log parser | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | R | ✓ | ✓ | ✓ | ✓ | ✓ |
| T03 · CSV with messy input | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | R | ✓ | ✓ | ✓ | ✓ | ✓ |
| T04 · Token-bucket rate limiter | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| T05 · Mini alerting DSL | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | R | ✓ | ✓ | ✓ | ✓ | ✗ |
✓ passed · ✗ DNF · R refused · highlight = winner by the chosen metric. Hover a task name for the full ranking.
Ranked by jobs completed first, always — then by the metric you pick. Cost and time are independent columns: the cheapest model is not the fastest.
Run through real Cerver sessions — metered, capped, transcript kept. The benchmark is a byproduct of the product.
Round 2 correction: round 1 showed cheap models "hitting a wall" — six of those DNFs were OUR bug (a transcript race made retries blind; the benchmark itself caught it). Fixed, re-run: 10 of 11 models now finish 5/5, and the cheapest model wins outright. This is why benchmarks need receipts.
claude-fable-5 (1/5): its failures are REFUSED verdicts — the model's API-side safety layer stochastically declines parser-writing tasks (it refused "parse a date string"; rich system prompts don't help). The same weights inside Claude Code complete all five tasks. Deployment context is part of the benchmark.