How Byte Pair Encoding Builds the Token Vocabulary Behind Language Models

Before a language model processes text, a tokenizer splits it into integer IDs drawn from a fixed vocabulary, and the model never directly sees letters or whole words. Byte pair encoding builds that vocabulary by starting from single characters and repeatedly merging the most frequent adjacent pair until a target size is reached. The article notes that vocabulary size is a tradeoff: larger vocabularies pack text into fewer tokens but add parameters, and because the vocabulary is learned from data, languages like Urdu end up costing far more tokens than English. The method originated as a 1994 data compression technique and was adapted for neural machine translation in 2016 by Sennrich, Haddow, and Birch to handle rare words. The authors implemented the algorithm from scratch on the paper's example dictionary to demonstrate how frequent words become single tokens while rarer ones split into pieces.