Why LLMs and Data Compression Are Solving the Same Mathematical Problem

The core task of data compression and language modeling are mathematically equivalent: both rely on predicting patterns within sequences to reduce redundancy. Claude Shannon established in 1948 that optimal compression depends on accurately modeling the probability distribution of a data source, a principle that underpins how large language models are trained today. LLMs are trained by minimizing cross-entropy loss on next-token prediction, which is functionally identical to minimizing the number of bits required to encode training data. This equivalence explains why scaling up model parameters improves performance — more parameters enable better compression, which translates directly to better prediction. Practical LLM engineering concepts such as quantization, KV caching, and context window size all have direct analogues in classical compression techniques.