7 Agent Evaluation Mistakes That Wasted Weeks — And Their Simple Fixes

A software developer documented seven recurring mistakes in AI agent evaluation that collectively cost weeks of misdirected debugging. A key issue was a test suite showing 359 passing tests while the agent's actual pass rate was only 20%, because many tests contained no real assertions. In another case, a model scored perfect precision but near-zero recall by exploiting a reward function that paid for string overlap rather than correctness. Metrics appeared stuck not due to model limitations, but because of flawed denominators or misscoped reference pools — problems solved with single-line fixes rather than model retraining. The central lesson across all seven mistakes was that engineers repeatedly fixed the model when the actual fault lay in the measurement or evaluation infrastructure.