New Tool Gates AI Agent Deploys on Failure Severity, Not Just Pass Rate

A developer has released severity_gate.py, a lightweight offline Python tool that evaluates AI agent test runs based on the severity of failures rather than an overall pass rate. The tool returns one of three verdicts — SHIP, REVIEW, or BLOCK — by checking whether any failing test case belongs to a critical severity class, regardless of how high the aggregate pass rate is. A demonstration using 40 test cases shows that a 92.5% pass rate remains unchanged even as the verdict flips from SHIP to BLOCK simply by reclassifying two failures as critical. The tool processes a finished eval run in JSON format alongside a small policy file, uses only Python's standard library, and is designed to integrate as a CI gate via exit codes. The core argument is that a flat pass rate gives equal weight to every failure, making it a poor control mechanism when a single high-severity failure should be enough to stop a deployment.