Developer Builds Go Library for Semantic LLM Caching to Cut Repeated Query Costs

A developer has released an open-source Go library designed to reduce cloud costs from repeated large language model (LLM) queries by combining deterministic hashing with vector similarity search. The tool addresses a common challenge enterprises face when scaling AI proofs-of-concept to production, where identical or near-identical user queries can generate significant API expenses. Key engineering hurdles included designing flexible cache key composition, managing concurrent background processes without memory leaks, and faithfully replaying streamed responses from cache. The library includes configurable options such as system prompt exclusion, async write-back workers, and TTL-based cleanup for both cached states and stream accumulators. Observability tooling via Prometheus metrics and a Grafana dashboard is also included, with the developer noting that the default 0.8 cosine similarity threshold may need tuning depending on real-world traffic patterns.