Cheapest per token is not cheapest per finished job, a 2,400-run benchmark finds

A 10-model benchmark of 40 real command-line tasks, run six times each across 2,400 runs by Arize and Fireworks, found that the cheapest model per token was not always cheapest per completed task. The researchers measured cost per successful task by counting all spending, including failed, retried and timed-out attempts, divided by the number of passing attempts. A low-cost model with a 33% pass rate still came out roughly twelve times cheaper per finished task than GPT-5.5, but it only covered a small set of tasks reliably. Results also varied by difficulty, with the gap between models widening considerably on hard tasks. The authors note the findings apply to agentic terminal work rather than other workloads, but they recommend that teams test cost per success on their own prompts and re-run comparisons as prices change.