Most AI Benchmarks Test Capability, Not Real-World Reliability, Dev Argues

A developer reviewing the GitHub 'awesome-evals' list has argued that most AI evaluation benchmarks measure what models can do in ideal conditions, not how reliably they perform in messy, real-world scenarios. Standard benchmarks typically present models with clean prompts and clear success criteria, while production agents face ambiguous instructions, failed tool calls, and unexpected errors. The author draws on months of running a self-hosted agent stack, where recurring failures stemmed from malformed tool arguments and models looping on unrecognized errors — issues standard evals rarely catch. They propose evaluating models on graceful degradation: how they handle retries, whether they seek clarification when stuck, and how they respond to bad tool outputs. As an interim solution, the developer is building a 'failure replay' pipeline that feeds real production failures back into a test harness to assess improvements in future model versions.