AI Firm Scores 99.95% on Memory Benchmark by Training Models on Test Data

A company achieved a near-perfect 99.95% score on LoCoMo, the leading benchmark for long-term conversational AI memory, by post-training memory directly into model weights using the same conversation set the benchmark evaluates. The team openly acknowledges the result does not prove their model is superior, but rather demonstrates the ceiling of parametric memory when a model is explicitly taught a corpus of conversations. Unlike the widely used retrieval-augmented generation (RAG) approach, baking memory into model weights eliminates recurring token costs, prevents cross-tenant data leakage, and enables fully offline deployment. However, the method carries real trade-offs, including slower updates, difficulty deleting specific facts under privacy regulations, and weaker generalization to unseen conversations. To address the benchmark's inability to separate recall from generalization, the team is proposing an extension called LoCoMo-Δ that withholds conversations from training to test true out-of-sample performance.