Building an LLM Red-Team Suite Reveals That Judging Harm Matters More Than Breaking Models

A developer built a red-team test suite to fire adversarial prompts at a local LLM-backed application, aiming to measure how often attacks succeed and whether the outputs are genuinely harmful. Using NVIDIA's open-source tool garak, the suite initially reported a 100% Attack Success Rate, yet only about 2% of responses contained anything actionable or dangerous. Even a smarter, content-aware detector dropped the rate to 73%, but real harm in those flagged replies remained close to zero, exposing a critical flaw in detectors that score how a reply looks rather than what it actually contains. The project found that accurately classifying harm requires human review, since automated metrics alone can report bypasses on batches where nothing harmful was produced. The developer concluded that structuring reliable datasets, defining clear harm criteria, and keeping a human in the loop is the hardest and most important part of AI red-teaming.