Zero-Parameter Document Cache Outperforms 1.4M-Parameter Transformer on Long Texts

A new experiment compared six language-prediction systems to measure how a simple document-frequency cache stacks up against a 1.43-million-parameter transformer with a fixed 64-token window. At short document lengths (around 60 tokens), the transformer holds a clear advantage, but the cache overtakes it somewhere between 60 and 250 tokens without any training, parameters, or significant compute cost. By 1,000 tokens, the cache leads by 0.064 top-1 accuracy — a 43% relative margin — because the transformer's fixed window cannot access most of a long document. When both mechanisms are combined, adding the full transformer on top of the cache contributes only 0.002 top-1 accuracy at L=1,000, suggesting the cache nearly replaces the neural model's contribution as documents grow. The findings highlight that document length is a resource the cache can exploit while the transformer structurally cannot.