Why AI Agents Need Real Security Controls, Not Just System Prompt Rules

Unlike chatbots that only produce text for humans to review, AI agents take real-world actions—sending emails, modifying records, or calling APIs—making errors or attacks far more consequential. Security researcher Simon Willis argues that system prompt instructions like 'never send an email without approval' are not true guardrails, since a language model treats all text in its context window equally and can be manipulated by malicious content embedded in documents. This vulnerability, known as prompt injection, allows attackers to slip instructions into inputs such as résumés or support emails, effectively hijacking the agent's legitimate credentials without any traditional breach. The attack pattern is called the 'confused deputy' problem: the agent acts on an attacker's command while using permissions your organization legitimately granted it. Willis concludes that the only reliable security boundary for AI agents is strict least-privilege access control—ensuring agents hold no more permissions than their specific task requires.