Real costs of running LLMs in production: caching, latency, and bill spikes

A developer building an AI-powered consumer app has detailed the hidden infrastructure challenges that emerge when large language model demos meet real-world traffic. Unlike tutorials that stop at the prototype stage, production deployments face compounding costs because models are billed per token and redundant queries can quickly inflate invoices. The author reduced API spending by 40–50% by implementing semantic caching using vector embeddings and cosine similarity, rather than exact text matching, so near-identical queries return stored answers instead of triggering new model calls. A subtle bug involving Gemini's embedding truncation—where shortened vectors lose unit-length normalization—silently degraded cache accuracy until manually fixed with renormalization. For vision-heavy features, perceptual hashing with Hamming distance tolerance was added to catch near-duplicate images before they reach the costlier embedding or vision-model calls.