AI Safety Benchmarks Can Fail When the Detector Itself Is Flawed

Developers at AgentSafeLabs discovered a critical flaw in their open-source AI security evaluation framework where the safety detector, not the language model, was producing incorrect results. In standard LLM safety testing, a classifier judges whether a model's response to an adversarial prompt is a refusal, compliance, or ambiguous — and those judgments feed directly into safety reports. During an investigation into apparently inconsistent model behavior, the team found the problem originated in the detector itself, including Unicode normalization failures. The most concerning issue was that a false PASS verdict goes unnoticed, unlike visible signs of uncertainty. The findings raise broader questions about how developers validate the classifiers they rely on to assess LLM safety.