Developer disables semantic cache after tests reveal it returns opposite answers with full confidence
A developer building a regulatory document retrieval system implemented a semantic cache using cosine similarity to avoid redundant model calls, setting a threshold of 0.92 to match near-identical questions. After a colleague challenged the safety of that threshold, a six-day investigation revealed the cache could return answers to questions with opposite meanings — such as 'patients with fever' versus 'patients without fever' — without any warning signal. Testing with 20 handcrafted Spanish administrative question pairs showed the highest adversarial similarity reached 0.9984, while the lowest genuine paraphrase scored only 0.7470. This meant no threshold could safely distinguish opposite-meaning questions from true paraphrases, making the cache fundamentally unsafe for rule-based queries where citation accuracy is critical. The developer disabled the cache by default and published a full write-up, noting that a bad cache hit is more dangerous than a bad search ranking because it bypasses the model entirely and shows no sign of failure.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in