Why Model Version Bumps Break Agent Workflows Without Failing Your Tests

Software teams using AI agents often find that upgrading a model version — even a minor one — silently changes workflow behavior while test pass rates remain high. Unlike gradual drift, this type of regression is a sharp discontinuity introduced by a known config change, such as a model bump or prompt edit, that hides in non-deterministic outputs. The recommended defense is a 'golden trace' system that compares new runs against pinned, known-good trajectories on objective, agent-proof axes. A three-tier evidence framework prioritizes externally verifiable facts — tool calls, schema validity, file existence — over statistical similarity metrics, with AI-as-judge reserved as an offline, non-blocking signal only. The key principle is that model-generated reasoning should never be graded solely by another model, as this creates a circular evaluation loop with no independent ground truth.