Python Semantic Cache Reduces LLM API Costs by Reusing Paraphrased Query Results

A developer has built a local semantic caching system in Python designed to reduce token costs when using free-tier large language model APIs. The cache converts user questions into vector embeddings and compares them against stored embeddings using Euclidean distance, returning a saved response when the distance falls below a set threshold. This approach addresses a key limitation of exact-match caching, which fails to recognise paraphrased questions as identical in intent and bills each variant as a separate API call. The system runs entirely on NumPy and JSON with no database required, storing past responses on disk for reuse across repetitive workloads such as support bots and content pipelines. By substituting a cheap embedding call for an expensive chat-model completion on repeated intents, the tool aims to stretch free-tier token allowances significantly further.