Build Your AI Agent's Eval Set First, Before Writing a Single Line of Agent Code

A developer series on building a support-ticket AI agent from scratch argues that evaluation test cases must be written before the agent itself is built. Creating evals after the agent works risks unconsciously designing tests around the agent's existing behavior, leaving silent failure cases undetected. The approach uses 21 structured test cases across three difficulty buckets — easy, hard, and edge — each checking outcome, tool-call trajectory, and policy compliance. Trajectory checking is highlighted as critical, since an agent can reach a correct final answer through a flawed or unsafe process. The series also notes that exact trajectory matching proved too rigid for non-deterministic LLM runs, a problem addressed later with ordered-subsequence matching.