Golden File Tests Are Failing AI Agents as Models Silently Drift

Software testing teams relying on frozen 'golden file' snapshots are increasingly finding them inadequate for validating AI agents, as underlying models change without notice and outputs shift unpredictably. Unlike traditional software, AI agents produce tool traces, side effects, and variable prose rather than a single stable reply, making static string comparisons unreliable. A three-layer testing approach is proposed: contract checks for JSON schema and required tools, effects checks measuring real-world outcomes, and a live critic model that reviews agent traces for plausible but flawed reasoning. The critic layer is identified as the most commonly skipped, often deferred until a customer demo or incident, which allows undertested agents to reach production. The recommended workflow involves capturing redacted traces, running deterministic contract gates first, and only then invoking a cheaper pinned critic model on a clean remote host.