Developer finds AI evaluation flaw: scorecards reward agreement, not better thinking

A developer comparing three AI coding assistants discovered that his checklist-based scoring system was inadvertently measuring which model agreed with his own thinking rather than which performed best. Any finding not on the checklist was automatically treated as a false positive, meaning no model could be rewarded for spotting something the evaluator had missed. After separating results into expected and unexpected findings — and having a third party assess the latter — two genuine overlooked issues emerged, each from a different model. The experiment also revealed that model rankings shifted depending on the task type, with larger performance gaps appearing in open-ended discovery tasks versus instruction-following ones. The developer concluded that choosing between AI tools may require task-specific evaluation, and that effective scorecards must account for unanticipated discoveries alongside predefined criteria.