Why AI Safety Stats Like '80 Quarantined Out of 1.2M' Can Mislead

A widely cited vendor safety metric — 1.2 million checks with 80 items quarantined — captures only blocked events, leaving the full picture incomplete. Security logging systems inherently record catches because they interrupt execution, while allowed requests pass through without comparable scrutiny, making false negatives structurally invisible. False positive rates can be calculated from quarantine reversals, but false negative rates require retaining individual allowed requests and later applying ground truth — data most systems never collect. Two guards catching the same absolute number of threats can differ vastly in actual effectiveness depending on the volume of bad requests each faced, a distinction no public catch count can reveal. In agentic AI systems, the problem deepens further, as allowed actions may only prove harmful days or weeks later, long after the context needed to re-evaluate them has been discarded.