Coding-Agent Evaluation Needs Negative Controls to Prevent Misleading Scores

An article advises developers to incorporate a negative-control slice when evaluating AI coding agents. This method prevents inflated scores from hidden test leaks, copied tasks, or broken assertions. The author recommends structuring evaluations into three distinct data slices: solvable tasks, negative controls, and canary tasks. Each task should be defined in a JSON object before testing to ensure methodological rigor. The goal is to maintain evaluation integrity and prevent unverified scores from being used for promotional purposes.