More Training Helps Transformers, But Cache Models Stay Competitive on Long Files

A new experiment tested whether training a small 2.45-million-parameter transformer on more data could close the performance gap with a zero-parameter count-cache model. Checkpoints saved at 500K, 2M, and 8M training tokens showed that the crossover point — where the transformer begins outperforming the cache — shifted from roughly 74 tokens to 492 tokens of document length as training data increased 16-fold. The cache model's accuracy remained constant across all training scales, since it never uses training data, serving as an internal control for the experiment. Extrapolating the observed rate, matching the cache on a 4,000-token file would require roughly 100–174 million tokens of in-domain code — far more than exists in typical per-repository settings. The findings suggest that while additional training does improve transformer performance, it cannot substitute for direct access to the current document context.