Semantic Caches Served Wrong Answers 30% of the Time in Ops Assistant Test

A developer experiment replayed 288 operations-related questions through a semantic cache using six embedding models and 50 different similarity thresholds. At a 0.80 similarity threshold with the bge-m3 model, the cache answered 32 percent of queries from memory but served incorrect answers roughly one in three times — for instance, giving git reset advice for a pushed commit when only a local undo had been cached. Raising the similarity threshold did not reliably fix the problem; between 0.88 and 0.92, wrong answers actually outnumbered correct ones among remaining cache hits. Across all models tested, the highest hit rate that produced zero wrong answers was just 0.7 percent, making the cache effectively useless at safe thresholds. Adding a small verification model reduced wrong answers but eliminated speed gains and made the overall setup more expensive than using no cache at all.