Fake Reviewer Note Bypassed AI Scam Detector, Exposing Prompt Injection Risk

A developer discovered that their AI-powered scam detector could be fooled by embedding a fake reviewer note inside a suspicious message, making the model approve content it should have flagged. Out of 19 adversarial test payloads attempted, only this one succeeded — and it required no elaborate jailbreak, just a plausible-sounding lie placed where the model expected trusted context. More concerning than the wrong verdict was the model's behavior afterward: it reproduced the attacker's fabricated claim as if it were its own reasoning, revealing that large language models lack any sense of text provenance. Attempts to fix the vulnerability through system prompt hardening backfired, making the detector overly cautious and degrading its accuracy on legitimate inputs. The fix that ultimately worked was a code-level pattern check applied to raw input before it ever reached the model — an approach the developer likens to the input validation and sanitization practices long advocated in application security.