How to Test AI Agents Using Pydantic Evals for Both Structure and Quality

Testing AI agents is fundamentally different from traditional software testing because large language models produce non-deterministic outputs and exhibit subjective qualities like empathy that cannot be captured by standard assertions. Two distinct challenges arise: inconsistent outputs and criteria that resist fixed-value checks. Pydantic Evals, the recommended evaluation library for Pydantic AI, addresses both problems through purpose-built evaluators. The IsInstance evaluator handles structural checks by verifying that an agent's response matches an expected output type, preserving classical assertion logic where possible. For qualitative assessment, LLMJudge uses a separate language model as a judge, scoring responses against a human-defined rubric to evaluate traits such as professionalism and empathy.