AI model benchmarks ignore cost — here's the math that actually matters

Popular AI model benchmarks typically rank outputs by quality and token usage, but almost never show what developers are actually charged. The true cost depends on a simple formula multiplying input and output tokens separately against a vendor's listed prices, and those two rates can differ by three to ten times. Using a sample coding task of 40,000 input and 12,000 output tokens, costs across leading models in mid-2026 range from roughly $0.056 to $0.50 per run — a ninefold spread no leaderboard displays. Choosing a cheaper model can backfire if it requires more retries and burns proportionally more tokens, potentially making it costlier than a pricier, more capable alternative. The author recommends logging input and output tokens separately per task, benchmarking top candidates on your own workloads, and recalculating costs quarterly as model prices shift frequently.