Voice AI Firm Finds Outbound Calls Had Half the Cache Hit Rate of Inbound Calls

A voice agent developer discovered a significant gap in LLM prompt cache performance between inbound and outbound calls — 70–75% hit rate versus just 20–30% — despite using identical code, prompts, and pre-warming strategies. Prefix caching, offered automatically by major AI providers, reuses previously processed prompt tokens to cut both cost and latency, but only when requests match byte-for-byte from the first token and exceed a minimum length of around 1,024 tokens. The team found that pre-warming successfully warmed the cache for individual calls but failed to sustain cache reuse across calls, which is where outbound performance suffered most. Analysis showed that outbound calls with only one LLM turn had roughly 24% cache hit rates, while those with four or more turns climbed to around 50%, confirming that in-call caching worked but cross-call reuse was breaking down. The investigation pointed to subtle prompt inconsistencies disrupting exact prefix matching, which silently invalidated cache hits before a single token could be reused.