DeepSeek-Linked Paper Proposes Engram, a Lookup Memory Module for Transformers

A new paper from a DeepSeek-linked team argues that large language models waste computation by re-deriving static facts on every query, and proposes Engram, a module that retrieves stored knowledge through a direct lookup. Scaled to 27 billion parameters, the authors report that it outperforms a mixture-of-experts baseline with the same parameter and compute budget across the benchmarks they tested. The largest gains appear on reasoning and code tasks rather than pure knowledge tests. The authors frame the design as a budgeting problem, finding a U-shaped trade-off between neural computation and static memory, with an optimal mix in between. Their Engram-27B model reallocates parameters from expert modules into a 5.7 billion parameter memory, which they say frees the network's early layers and attention capacity for harder work.