Semantic and exact-match caching cuts LLM costs beyond provider routing

A software team found that caching language model responses was a more impactful cost-reduction lever than routing traffic across providers. By hashing full requests and returning stored completions on exact matches, they eliminated redundant model calls on high-traffic endpoints where prompts were far less unique than assumed. They also implemented semantic caching using vector embeddings, reusing prior completions when a new query exceeded a 0.92 similarity threshold, though this was limited to deterministic tasks like classification and extraction. The combined approach achieved a cache hit rate of around 35% on cached endpoints, reduced p95 latency to under 50ms on cache hits, and pushed overall cost savings well beyond the initial 40% gained from routing alone. The team emphasized tracking hit rate over time to detect prompt drift and ensuring embedding lookups remain cheaper than the model calls they replace.