Study Finds N-gram Count Tables Lack Memory Locality, Defying Earlier Assumption

A software developer re-examined a core assumption in a series of language model experiments: that count tables used for next-token prediction would behave like mixture-of-experts models, keeping only a small 'hot set' of data resident in memory. To test this, the researcher rebuilt the tables as memory-mappable, pageable arrays and verified they produced identical accuracy results to the original in-memory versions. Measurements across eight real code files showed that working set memory kept growing throughout roughly 2,700 completion positions per file, with seven of eight files still expanding at the end of the session. A bounded cache test confirmed the problem, with hit rates plateauing at only 74.4% even when 16 MB was held resident, and dropping to 47.3% at 1 MB. The findings show that while code tokens repeat frequently, the count table is indexed by context rather than raw tokens, meaning locality at the token level does not translate into locality at the lookup level.