Model Eval Tool Exposes Risk of Conflating Errors With Wrong Answers

Model-drift, an open-source tool, grades AI models weekly using an exact-match scoring system with no LLM judge, making score changes reliable indicators of real performance shifts. However, a design flaw means absent answers — caused by rate limits, timeouts, refusals, or truncations — score the same as wrong ones. The developer discovered this firsthand when a model appeared to crash from 69% to 3% accuracy overnight, only to find that 34 of 35 calls had failed due to API rate limiting. To address this, the tool excludes a run's accuracy data only when overall reliability drops below 50%, rather than penalising individual failed calls. This approach prevents skewed results, since harder prompts are more likely to trigger failures, meaning blanket exclusions would artificially inflate accuracy scores on the most challenging tasks.