Why Top Leaderboard Coding Agents Can Fail on Your Actual Codebase

A developer discovered that selecting a coding agent based solely on public benchmark scores led to poor real-world results, with the tool producing diffs that broke files and failed internal review standards. Public benchmarks were found to be unreliable because models may have been trained on the test data, and the controlled benchmark environment bears little resemblance to messy real-world codebases. The team's internal libraries, coding conventions, and stricter review criteria meant that textbook-correct code still got rejected. To fix the evaluation process, the developer built a custom replay test using the team's last 50 merged pull requests, rolling back the repo and letting agents attempt the same tasks from scratch. Scoring was based on whether the agent's output would have passed the team's actual review process, not on textual similarity to the original diff.