Developer finds dead-code bug in own AI security scanner while probing LLM vulnerabilities

A developer built AgentProbe, a tool that fires 49 known attack prompts across 8 categories at AI models to test their resistance to prompt injection, currently ranked the top security risk for LLM applications by OWASP. While building the scanner, the developer discovered a logic bug where a custom 'hedge-then-comply' detector always returned a confidence score of 1, but the escalation threshold was set at 2 or higher, meaning the detector's results were silently discarded every time. As a result, every case the cheap keyword detector was meant to handle was unnecessarily escalated to a more expensive LLM-as-judge call, wasting resources and creating a single point of failure. The bug went unnoticed because the LLM judge independently caught the same patterns, masking the fact that the keyword stage was effectively dead code as a decision-maker. The incident highlights a broader concern in AI evaluation: LLM-as-judge systems are widely used in safety benchmarks and model leaderboards, yet the reliability of the judge model itself is rarely verified.