Agent benchmarks measure memory-less models, missing a key performance variable

A critique published on DEV Community argues that AI coding-agent benchmarks are structurally flawed because they always start models with empty memory, ignoring how accumulated project context affects real-world performance. The argument uses Engrim, an open-source SQLite-based persistent memory engine for tools like Claude Code and Cursor, as evidence that cross-session state is a meaningful and underexplored performance axis. Current evals deliberately reset memory between runs to ensure reproducibility, but this means the benchmarked agent bears little resemblance to one that has ingested months of project decisions. The author proposes a fix that preserves reproducibility: run each task under three conditions — empty context, small curated memory, and large memory — and measure how scores shift across conditions. Until benchmarks account for persistent state, their results represent only a performance floor, not a reliable predictor of production behavior.