Hybrid Search and Local Reranker Boosts Agent Memory Accuracy from 63% to 80%

A developer benchmarked a homelab AI agent memory stack using LoCoMo, a long-term conversational memory benchmark that tests recall across multi-session dialogues. The initial vector-only retrieval system scored 63%, with failures concentrated on lexically specific queries involving ticket numbers, dates, and proper nouns rather than semantic understanding. Switching to a higher-ranked embedding model yielded only a 2-percentage-point improvement, revealing that the core problem was not semantic but lexical. Adding a sparse retriever alongside the dense vector search, combined with a reranker running on existing hardware, pushed the score to 80% at a latency cost of roughly 40 milliseconds per query. The author notes that a flawed memory stack fails silently by feeding the model incorrect context, making it a more dangerous failure mode than a system outage.