AI Coding Agent Benchmarks Are Misleading Without Retry and Time Caps

Reported pass rates for AI coding agents are often incomplete because they omit critical controls such as retry limits and wall-clock time ceilings, making comparisons between tools unreliable. Two agents can show identical success rates while one solved tasks in a single attempt and another silently looped through multiple repair cycles. A proposed evaluation framework calls for sealed task packs with frozen prompt templates, hidden test digests, explicit retry ceilings, and tool-trace identifiers as mandatory dataset fields. The framework also defines a small metric set — including a boolean pass result, attempts used, elapsed time, and a trace file hash — to detect hidden rework and incomplete runs. Without these standardized controls, published benchmark percentages resemble marketing figures rather than reproducible experimental evidence.