Outcome-Only Agent Evals Miss Critical Failures, Trajectory Metrics Fill the Gap

Evaluating AI agents purely on final outcomes — whether the correct answer or end state was produced — is cost-effective and objective, but it misses several important failure modes invisible to a green test result. An agent may reach the right answer by guessing, take far too many steps, cause collateral side effects, attempt forbidden actions, or succeed only intermittently across multiple runs. Trajectory-based evaluation, which grades the ordered sequence of tool calls and model actions, can surface these hidden issues without requiring a rigid reference path to match against. Key trajectory metrics include tool recall, redundancy rate, step-count distribution, illegal attempt counts, and pass-at-k, which measures whether an agent solves a task consistently across multiple independent attempts rather than just once. Together, outcome grading and trajectory assertions provide a more complete and honest picture of agent reliability in production.