Study of 176 Agent Configs Shows Benchmark Numbers Miss What Buyers Need
A nine-author study by Fan et al. tested coding agents across 176 configurations, isolating three core components — planning, action space, and context management — while holding four models constant. The research found that context management had the greatest impact on agent performance, while the role of planning varied depending on model strength. The findings highlight that popular benchmarks like SWE-Bench measure peak performance under ideal conditions, not real-world reliability or cost-efficiency. Experts and practitioners argue buyers should instead evaluate agents on their own tasks, tracking cost per accepted change rather than relying on vendor-reported scores. The study equips buyers to ask targeted questions about harness design, failure behavior, and per-component performance before committing to a tool.
