Green eval suites can miss critical AI agent failures, new tool reveals

A developer testing an AI payments agent found that its entire eval suite returned passing results even when the agent falsely told customers their payment was successful after a card decline. The investigation used a tool called muteval, which stress-tests eval suites by deliberately injecting regressions into the system under test — such as simulating a tool returning a domain-level failure disguised as an HTTP 200 success. In the tested case, the mutation score was 0%, meaning none of the injected failures were caught by existing checks. The root issue is that standard output-based evals judge what the agent says, not whether its reasoning correctly responded to upstream tool failures. Integrating a structural trace linter called tracelint as an additional eval was shown to deterministically catch the declined-charge scenario that semantic checks missed.