Why Comparing AI Agent Pass Rates Without Token Budgets Is Misleading

A software engineering analysis argues that pass rates alone are an unreliable benchmark for evaluating AI coding agents when the token and tool-call budgets differ between runs. An agent that retries repeatedly until hitting a cap may outperform one given a single attempt, yet both results get averaged as if conditions were equal. The proposed protocol requires declaring a fixed budget before any model call, logging every attempt in a JSONL ledger, and publishing three metrics together: pass rate, truncation rate, and tokens-per-success. Task files must be hashed with SHA-256 so any fixture change invalidates the run ID, ensuring reproducibility. The author frames this as a methodological standard rather than a vendor comparison, noting that omitting budget data turns an engineering review into what they call 'a vibe with a percentage sign.'