&#8592; All posts Which Model Tops Our Leaderboard? How the LLMs did in our realworld tests. Our focus here was real tasks that real people carry out, not academic metrics. We focus on single tasks to simplify the assessment. An agentic flow is ultimately a series of such tasks. Think of these like unit tests for the agent. We made them cheap enough to run so that even the whole suite costs just $30. See every task and each model's actual answer, or compare two models head to head &#8594; Pass rate Rubric quality Cost / task Latency (TTFT) The overall score is the pass rate across my 28 realworld tasks. As we only had a limited number of trials there is a wide Wilson interval — the whiskers on the chart. Summary of results: click a column to sort by your chosen metric. # Model Pass rate (95% CI) ▼ Rubric /10 ▼ Security ▼ Median TTFT ▼ Run cost ▼ Cost / task ▼ Four caveats on how these numbers were produced 1 Rubric scored retroactively (14 Jul 2026) by fable-5 against the saved answer text, through the harness's own run_rubric path — same blind prompt and criteria as every other row. 2 fable-5's 9.3 is self-judged — the judge scoring its own answers. Its source run's judge-bias matrix shows it rating itself 9.3 versus 8.6–8.7 for the models it judges independently. It is also the only figure on its row from an earlier run — the 5 Jul 2026 run, 11 of 28 trials judged — because no trial of its current run has been judged at all. Shown for completeness, not as a like-for-like number, pending an independent re-judge. 3 Recipe-checker false-positive. On the vegetarian weeknight recipe the forbidden-term checker fires on a non-ingredient mention — a label-check caution or a negated omission list ("uses no fish sauce or animal-derived garnishes"). All three recipes are genuinely meat-free, so gpt-5.5, sonnet-5 and fable-5 are scored as passing that task here. No task or checker was edited. 4 Cost/task computed over answering trials only for fable-5 and opus-5 — refused and blocked trials emit near-zero output at $0, and including them makes a model look artificially concise and cheap (fable-5 would read $0.0481/trial; opus-5 $0.0597). opus-5's headline run cost of $1.67 is the true all-trials total: the blocked trials were billed $0. No other model on the board has refusals. The lap, corner by corner # The most recently added models appear first, with the latest test date shown under each. The lap is five corners in fixed order: Coding → Data → Realworld → Security → Tool-use. A corner's colour is that model's pass rate in that category. Green is good — it means 85%+ success. For models that can do it all, look for all green. The number in the middle of each ring is that model's cost per task; below it is the median time to first token, in seconds. Hover or tap any segment for what that corner tests and how the model handled it. Clean corner (>85%) Ragged (60–85%) Off the track (<60%) ★ Our pick — the All-Star champion, the desert island model Our pick for a low-cost workhorse ⚡ Our pick for the fastest reply See the exact numbers by category Cells below 60% are flagged red and 60–85% amber — coding, data and tool-use are the harness floor, so the race is decided in realworld and security. Model Coding Data Realworld Security Tool-use What Do the Results Actually Tell You? If you only run one model, run glm-5.3 glm-5.3 is the first model on the board to clear all five corners — coding, data development, realworld, security and tasks — at 100%. It backs that with a 9.3 rubric, third-highest on the board, and $0.28 for the lap. The one cost is patience — a 16.3-second median time-to-first-token. gpt-5.5 is the faster alternative at 13.2s, with the same 100% security but an 89% realworld corner and $1.43 for the lap. Fable failed to complete a single lap fable-5 is joint-bottom at 79% because it refused to do 5 of the tasks. It performed well on what it completed, but even it thought kimi-k3 was giving better answers. You'll need a fallback model if you're using Fable. opus-5 hit the same wall — four benign coding-debug-* tasks blocked before a token was generated, on an overlapping set of tasks — so Anthropic's classifier looks like it sits across the whole series 5 line, not just Fable. See the full refusal breakdown for what's actually going on. Luna is the very cheapest workhorse gpt-5.6-luna costs $0.064 for the full lap, or $0.0023 per task, with a 5.3-second median TTFT. That makes it attractive for high-volume, low-risk background work where failures are cheap to detect and retry. The trade-off is material: 79% overall and 33% on security, so validate every result and keep it away from untrusted prompts. haiku-4-5 is the higher-pass alternative at $0.0044 per task, 96% overall and a 0.9-second TTFT. deepseek-v4-pro is nominally cheaper still at $0.0029 per task for the same 96% pass rate, but its 40.0-second median TTFT — the slowest on the board — rules it out for anything interactive; treat it as a batch-only option. The mystery guest sets the fastest quality lap kimi-k3 still tops the rubric at 9.5 — judged independently by fable-5 — with a 96% pass rate, though opus-5's 9.4 now runs it close on quality at a third of the wait. The catch is patience: a 26.4-second median time-to-first-token, second slowest on the board behind deepseek-v4-pro's 40.0s, and a 75% wobble on data development tasks, its only weak corner. Not suitable for interactive applications. Three cars failed the crash test The gpt-5.6 line is quick, but it has a safety problem. gpt-5.6-luna, gpt-5.6-terra and gpt-5.6-sol emitted the jailbreak canary in 11 of 12 jailbreak cells (33–50% security pass) — make sure you protect in your harness, and apply more careful Red teaming if using these models. The Claude trio went 6/6 clean, as did gpt-5.5. A safety filter can look exactly like a bad lap opus-5 posts the best rubric on the default panel at 9.4 and 100% on both realworld and security — then shows 43% on coding. That cell is not its debugging ability: four benign coding-debug-* tasks were blocked by a provider-side classifier before a single token was generated, on an overlapping set of tasks to the ones already blocked on fable-5. Two Anthropic-family models now hit the same filter, so treat it as a measurement hazard rather than a model quirk — and note opus-5 was also penalised twice for flagging an attack it had successfully resisted. How Is the Ed-o-meter Scored? Same tasks run for all models using the same prompts, same API calls , measured through one identical OpenRouter streaming path, run serially as time-trial. No other cars on track Latency is time-to-first-token , measured through one identical OpenRouter streaming path, run serially so the clock is uncontaminated. Wall-clock is recorded alongside. Checkers are binary and automated. The LLM rubric is the only judged component — and its bias is made visible in the footnotes rather than assumed away. Effort and reasoning settings are pinned in models.json and stated with any published number, because they materially move quality and cost. Refusals are recorded, not hidden. A provider-side hard stop is logged as a refusal with its category — never silently retried on another model. Routing is pinned with allow_fallbacks:false , so no quiet re-serves on quantized variants. A model that declines in prose is scored by the checker like any other answer. Harness, tasks and checkers are open source at Featherbench (MIT). Clone it and run the lap yourself, or request a new model via GitHub issue . See all 28 tasks Coding (7 · Python) CSV dedupe — small, well-specified task with a deterministic unit-test checker Debug billing date — fix a month/day-overflow date bug without regressing the working cases Debug money split — split integer pennies N ways so shares sum exactly and stay fair Debug mutable default — fix the classic mutable-default-argument bug Debug pagination — fix an off-by-one page-count bug Log parsing — parse logs