Developer scraps 300-model AI benchmark after finding most models now perform comparably

A developer running the 'Agent Autopsy' series spent months benchmarking over 300 large language models across ten real-world coding tasks, including file operations, SQL queries, and error recovery. The entire dataset was built using OpenRouter and local hardware at a total cost of less than a typical takeaway meal. Over just a few weeks, top scores jumped from 50% to a 90% floor at sub-penny prices, making model-quality comparisons increasingly redundant. The author ultimately concluded that the more useful question had shifted from which model performs best to how little scaffolding is needed — a question a leaderboard cannot answer. With no evidence that anyone was using the data to make real decisions, and models updating too frequently to keep rankings accurate, the benchmark was deliberately shut down.