Semantic Cache Hit Rates Can Hide Wrong Answers, Seven Tests Show

A semantic cache reuses stored LLM responses when a new query is similar enough to a past one, and its hit rate alone does not show whether reused answers actually fit the new question. In a test using Actian VectorAI DB at a 0.80 similarity threshold, the cache hit 79.2% of lookups, but 6.4% of those hits returned wrong answers, some with similarity scores above 0.95. The author ran seven tests covering threshold calibration, negation, changed numbers and dates, tenant and intent boundaries, local verification, query length, and stale cached answers. Each test lowered the false-positive rate, though often at the cost of hit rate, and the results are presented as controlled measurements rather than production guarantees. The experiments used 3,075 Banking77 questions as the cache and 770 held-out queries for lookups, with a 5% false-positive target.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in