Benchmark-Driven Loop Reshapes IWE's Local Markdown Agent Memory System

Developers behind IWE, a local-first knowledge tool that stores memories as plain markdown files in a graph structure, built a benchmark to test whether their system could replace cloud-based AI memory layers. Using the LOCOMO dataset — a standard evaluation corpus featuring long fictional multi-session conversations — they repeatedly measured, failed, and rebuilt their search and editing tools based on what the benchmark revealed. The iterative process led to a redesigned search engine, a new block-level editing language, and the abandonment of several original design rules. The team also retracted their own best-performing result after scrutiny, ultimately settling on a low-cost curation model with mechanical safeguards. The final system achieved 96% of a hand-built performance ceiling in a single retrieval call, while a simple grep-based baseline still held its own on raw accuracy.