Eval gate passed at 97% because the agent's own answers were used as expected results

A support agent approved 37 refunds between $200 and $250 without human review over six weeks, violating a policy requiring approval above $200. A typo in a prompt edit had raised the limit in the agent's config, and the eval gate still passed 11 releases in a row. The gate missed the error because many expected answers had been recorded from the agent's own output, so the agent and the eval agreed with each other while both were wrong. The author concludes that expected answers must come from the policy or other source of truth rather than the agent, and that evals should not share the agent's config. After rewriting the cases from policy, seven previously encoded cases were found wrong and four recent releases failed, which had previously been considered good. The rebuilt set of 480 cases draws on production samples, policy boundaries, incidents, adversarial inputs and failure shapes.