Security Test Checklist for AI Agents That Can Call External Tools

As LLM-powered agents gain the ability to call real-world tools like refund systems or email services, traditional jailbreak testing is no longer sufficient to ensure safety. Security engineers are advised to prioritize testing tools that write data, run under broad service accounts, and lack downstream enforcement limits. A key principle is to verify actual system state changes — such as database records — rather than relying on the agent's text response to determine whether an attack succeeded. Indirect prompt injection, where malicious instructions are embedded in content the agent reads such as emails or PDFs, poses a significant risk; research found GPT-4 agents followed injected instructions roughly 24% of the time. Because agents are non-deterministic, high-impact adversarial test cases should be run multiple times and tracked by success rate rather than treated as a single pass-or-fail check.