Developer's Custom AppSec LLM Prompt Found 4 Real Gaps After Filtering AI Noise

A developer nearly filed 23 unverified vulnerability reports against open-source projects after running a self-built AppSec prompt across several repositories, with the LLM flagging XSS, CORS, CSP, and suspicious SVG issues in bulk. The author paused to question whether the volume of findings would be taken seriously by maintainers, which prompted a rethink of how the prompt was structured. The key shift was writing a strict contract before any code review — defining what the model was permitted to assert, anchored to OWASP ASVS 5.0 requirement IDs rather than vague best practices. Discipline rules were added to separate findings from observations and hypotheses, and to prevent the model from mixing discovery with fixes in the same session. The author concludes that static scanners like Semgrep and CodeQL remain essential for pattern-based detection, while LLM-driven hunting adds value only for semantic gaps — absent controls that no regex can identify.