AI Voice Agents Face Security Risks From What They Can Do, Not Just What They Say

Security vulnerabilities in AI voice agents stem not from jailbreaking or prompt manipulation, but from the real-world actions these agents can perform, such as reading CRM data, booking appointments, and sending texts. Because the entry point is simply a phone number with no login or verification, anyone who calls can potentially trigger these actions. A developer building agents for clinics and other businesses found that tools often accept caller-supplied parameters without verifying the caller's identity, allowing strangers to access other people's records. Caller ID spoofing and shared phone numbers make phone numbers a weak form of authentication, so the developer now splits access by consequence — requiring an additional verification step like a date of birth for any sensitive data or record changes. System prompt instructions are described as useful for shaping normal behaviour but ineffective as a security control, much like client-side form validation alone cannot protect a database.