Why grading an AI agent's final answer misleads teams, and how to evaluate it properly

A practical guide argues that judging an AI agent only by its final message hides failures, since an agent can reach a correct outcome through faulty tool use. It proposes a layered set of eight metrics that move from outcomes inward, including tool selection and parameter accuracy. The guide stresses consistency across repeated runs, noting that an agent succeeding 75% of the time clears three consecutive runs only about 42% of the time. It also recommends calibrating LLM-as-a-judge graders against human-labeled examples, mitigating known judge biases such as position and length preference, and converting real production failures into regression tests.