Immutable Task Ledgers Are Key to Honest AI Agent Benchmarking

Credible evaluation of coding agents requires that datasets be treated as versioned, immutable artifacts with hashes, licenses, and difficulty labels assigned before any model is run. Teams that skip this freezing step risk publishing scores that cannot be reproduced, effectively presenting marketing figures as rigorous benchmarks. A proposed methodology uses cryptographically hashed JSON Lines records to lock task definitions, preventing retroactive changes that could skew rankings. Controls such as cluster identifiers help ensure near-duplicate bugs are not counted as separate victories, while license metadata ensures scores are built only on legally clearable code. A simple Python gate can enforce ledger integrity by refusing to score any run whose dataset digest does not match a pre-committed value.