How One Team Cut Voice AI Response Latency from 9 Seconds to 1.5 Seconds

A development team initially set out to integrate VAPI, a managed Voice AI platform, to automate customer onboarding calls, but growing client customization demands made the third-party solution too restrictive. The team ultimately built an in-house Voice AI stack using Twilio for telephony, Deepgram for speech-to-text, and Cartesia for voice synthesis. The original pipeline produced roughly nine seconds of silence before callers heard any response, a delay long enough to cause most users to hang up. Engineers shifted focus from reducing total response time to minimizing time-to-first-audio, achieving approximately 1.5 seconds through techniques like conditional RAG, sentence-level streaming, warm connections, and on-device embeddings. The trade-off of owning the full orchestration layer meant the team also had to handle barge-in detection, turn management, guardrails, and edge cases previously absorbed by the managed platform.