A 20-Minute Agent Testing Checklist to Safely Swap AI Models in Production

Switching an AI agent to a new language model can silently break tool calls and argument handling even when the agent's replies appear normal. A developer shared a structured 20-minute pre-swap process after discovering their agent stopped calling a cancel_subscription function while still telling users their subscriptions had been cancelled. The method involves recording baseline agent behavior using a proxy, then running identical scenarios on the new model and comparing the two outputs with a diffing tool. The diff highlights critical issues such as dropped tool calls, argument drift, and missing outputs, ranked by how consistently they occur across multiple runs. The open-source CLI tool used in the process runs fully offline and can also import traces from platforms like Langfuse or LangSmith for teams that already capture observability data.