The task being measured
An agent that plans dog walks — and can't fake it
Give it a starting point and a few dogs, each with an address, a walk length, and constraints — a comfortable temperature band, a morning or afternoon window, hills to avoid, medication to administer. A small language model drives the whole thing: it geocodes the addresses, solves the visiting order over real streets, checks the weather for each dog's actual walk window, and returns a schedule with a per-dog safety verdict. It does that by calling four tools and relaying what they return.
Street addresses → coordinates (Nominatim / OpenStreetMap).
Hourly forecast for a place and time window → a deterministic OK / CAUTION / SHORTEN / DO-NOT-WALK verdict.
Elevation around a dog's home → whether the hills exceed that dog's stated tolerance.
Real walking distances (OpenRouteService + OR-Tools) → the visiting order and a timed schedule; returns feasible:false when the windows can't all be met.
The model is the least-trusted part of the system, and the measurement only works because it's boxed in. A referee validates every tool call against its schema before it runs. An auditor — plain code, no AI — refuses a finished plan unless every dog's walk has a real weather check at its own location. And the plan is submitted through a structured submit_plan whose validated arguments are the answer, never parsed from prose. Those checks are what make a "pass" mean something — the deterministic finish line a model can be measured against. Watch it run live →
The verdict
A $0.0004 model ties the frontier — and the model that looked worst was mostly our own bug.
Two models finished all 35 runs, and the cheaper one — 66× cheaper — is the new default. The bigger surprise came from the model that placed last in the first sweep: most of its failures weren't the model at all, but a bug in this harness. Fixing it moved the old default from 77% to 97%.
Perfect, fastest of the cheap tier, and the least expensive model in the sweep. Now the site default.
Also flawless — and it alone burned 70% of the whole sweep's cost. The reliability is free; the certainty is not.
It placed last in the first sweep. The harness traced the losses to our own referee rejecting max_relief_m: 0 — the model's shorthand for "no hill limit." We fixed that; the re-run scores 97%, one truncated response short of perfect. The bug was ours, not the model's. (Still not the default — ling is 18× cheaper.)
Reliability
Pass rate, with the doubt drawn in
Five runs can't prove 100%. Each dot is the observed pass rate; the whisker is the 95% Wilson interval — the honest range the true rate could sit in given only 35 samples. Note how wide even a perfect score's floor is, and how far the bottom two models' intervals reach.
Overall pass rate by model
35 runs per model · dot = observed rate, whisker = 95% CI
Cost vs. reliability
The frontier tax
Plot what a plan costs against how often it lands and the trade-off stops being abstract. The upper-left is the prize: reliable and cheap. ling-3.0-flash sits there almost alone. haiku-4.5 matches its reliability two orders of magnitude to the right.
Median cost per plan × pass rate
cost axis is logarithmic · each point is one model
Where they break
One scenario carries most of the difficulty
Averages hide the shape of failure. Out of five runs per cell, the strong models are near-perfect except on one column: full-house — a single dog wearing every attribute at once (comfort band, meds, a morning window, prep time, and hills). mercury-2.5 is the exception, thin across the board. And the first sweep's headline — the old default going 0 for 5 on Lakeview — is gone: that was the referee bug, now fixed, which is why qwen3-8b's row is almost solid green.
How they fail
Not all failures are the model's fault
Eighteen runs didn't pass, and lumping them into one number lies — they aren't the same kind of thing. The largest category in the first sweep, the referee being too strict, is gone entirely; fixing it is what recovered qwen3-8b. What's left is mostly infrastructure — a flaky provider, and long tool-call JSON that got truncated — plus a couple of genuine model mistakes.
The one that matters: fabricated feasibility
Two runs, both from mercury-2.5, declared the morning-overbook roster solvable. It isn't: four hour-long walks that must all start before noon cannot be fit into one outing. The model called the route tool but dropped the walk-window constraint, so the deterministic tool saw an easy route and agreed.
The in-loop auditor is blind here — it can only check the constraints the model actually passes. The measurement caught it anyway, because the harness knows the right answer independently. That is the whole thesis: you can measure an agent honestly only when the task has a checkable finish line the model can't talk its way past.
Reading the caveats honestly
mercury-2.5's 71% is mostly an availability story: 8 of its losses were the provider returning no completion at all, not bad planning. gpt-oss-120b's dip to 86% is the same shape — a burst of malformed and truncated tool-call JSON on the heavier scenarios, not its planning; the float(None) bug that cost it two runs in the first sweep is fixed and gone. And the biggest correction is qwen3-8b: its first-sweep 77% was almost entirely our referee rejecting max_relief_m: 0. That's fixed, and the re-run confirms it — 34/35, a twenty-point jump. The lesson points back at the harness: a grader this strict catches your own bugs before it catches the model's.
How it was measured
A grader that can't be charmed
The task is dog-walk route planning: geocode addresses, solve the visiting order over real streets, check weather for each dog's actual walk window, respect terrain, medication, and comfort constraints, and finish with a structured plan. The numbers here combine the 2026-09-14 sweep with a 2026-09-15 re-run of the two models whose failures the harness traced to bugs since fixed. Try it live →
01 Deterministic grade
A run passes iff it reaches a schema-validated submit_plan whose feasible flag matches the scenario's known answer. No model judges another. All seven scenarios' answers were checked against the solver before the sweep.
02 Real cost & latency
Dollars, not token estimates. Every model round's OpenRouter usage — prompt and completion tokens plus the actual charge — is summed per plan, so cost-per-run is measured, not modeled.
03 Five runs, real intervals
Each of the 42 model×scenario cells runs five times. Pass rates carry a Wilson score interval, so "5/5" and "50/50" are never mistaken for the same confidence.
04 Failures kept whole
Every failing run's full transcript is archived and sorted into a taxonomy from the event stream — veto_livelock, schema_thrash, prose_stall, fabricated_feasible, false_infeasible, backend_error, timeout. The rejects are the content.