RAG Prompt Injection Test Shows Structural Guards Beat Model Cleverness

A developer experiment demonstrated how a malicious instruction embedded in a help-centre article can manipulate an LLM-powered support agent into proposing an unauthorised refund when a customer asks an unrelated question. In the first test scenario, the injected command targeted another customer's order, which was automatically blocked because session-scoped lookups only permit access to the authenticated user's own orders. A harder scenario used the victim's own eligible order, and while the agent did propose a refund it had no reason to, the action was queued for human approval rather than executed automatically. The key finding is that high-risk actions like refunds were classified as requiring human sign-off, meaning the system's structural controls — not the model's judgment — prevented financial harm. The author argues that security cannot rely on the AI model detecting or resisting manipulation; instead, deterministic boundaries and risk-gated approvals must enforce safety regardless of what the model proposes.