Developer builds AI guardrail benchmark designed to expose failures, not hide them

A developer has published a deliberately self-critical benchmarking methodology for Doberman, an AI agent guardrail system, aiming to report honest results rather than flattering ones. The framework scores the tool against a 158-row labeled test corpus across seven attack categories, tracking not just attack success rates but also false positives and the impact of human fatigue on outcomes. A key distinction is drawn between hard blocks and human-authorization prompts, with the latter classified as a 'leash, not a wall' since a human who approves a flagged request leaves the system unprotected. Results show that in balanced mode, the guardrail achieves a 77% true-positive rate overall, but only 8% of mitigations are hard stops — the rest rely on a human saying no. Documented gaps, such as a 0% detection rate for natural-language injections lacking a recognizable command shape, are listed prominently rather than omitted from the published benchmark.