Why a Perfectly Flat Performance Score Is a Sign Your Measurement Is Broken

A developer running AI agent systems discovered that a 0% skill evaluation score persisting across every recorded run was not evidence of poor performance, but of a broken evaluation harness using the wrong authentication token, causing every API call to return a 401 error and be silently marked as a failure. The core insight is that genuinely underperforming systems produce variable scores, because they still interact with a changing world, while a perfectly flat or extreme number typically signals a fault in the measurement pipeline itself, not in the system being measured. The same pattern appeared across three industries: a hardcoded dashboard figure in advertising that went unquestioned for two years, a generic error message masking distinct failures in agent operations, and a medical-billing agreement metric that appeared alarmingly low at 44.7% only because two sides of the comparison used different scoring units. Once the units mismatch was corrected, the billing engine's accuracy was recalculated at 92.0% for classification and 90.7% for action against a golden test set. The author now treats an unchanging metric as a red flag, arguing that a healthy measurement should vary over time as it stays in genuine contact with the system it monitors.