How a 1994 Compression Algorithm Became the Backbone of LLM Tokenization

Modern large language models rely on a tokenization method rooted in Byte Pair Encoding (BPE), a data-compression algorithm originally developed by programmer Philip Gage in 1994. Tokenization is the process of breaking text into smaller units called tokens before a model processes them, and the way this split occurs affects input length, inference cost, vocabulary size, and multilingual performance. BPE works by repeatedly merging the most frequently co-occurring byte sequences in a training corpus, striking a balance between character-level and word-level approaches. Unlike word tokenizers, BPE can handle rare or invented words by decomposing them into known sub-units, without requiring an infinitely large vocabulary. The algorithm gained widespread adoption in neural machine translation before becoming central to tokenizers used in models like GPT-2 and its successors.