Developer Builds LLM Vulnerability Scanner That Judges Bugs, Not Just Finds Them

A developer building an open-source LLM-based vulnerability scanner deliberately designed it to generate false positives, using them to test how well the AI judges code safety rather than just flagging suspicious patterns. The core insight is that traditional rule-based scanners like Semgrep follow data flow from source to sink but cannot understand semantic meaning, causing them to flag safe, validated code as vulnerable. Meanwhile, using an LLM alone to scan codebases introduces inconsistency, poor search reliability, and hallucination risks. The proposed architecture separates responsibilities: deterministic tools such as Joern handle all candidate detection via fixed taint queries, while the LLM evaluates only the shortlisted snippets one at a time, aided by relevant CVE examples retrieved via RAG. This division of labor aims to combine the reproducibility of static analysis with the contextual reasoning of large language models.