Why Tokenizers Are a Fixed Part of Every Language Model, Not Just Preprocessing

Language models do not process raw text — they receive sequences of integers produced by a tokenizer, a component that is permanently fixed at the time of training. Replacing the tokenizer after training renders the model's learned weights effectively useless, even though they remain intact. Tokenizers use subword encoding, a technique adapted from a 1994 compression algorithm, to balance vocabulary size against sequence length. Design choices baked into the tokenizer — such as how digit sequences are split — directly affect what arithmetic or reasoning a model can perform. This article, the first in a 12-part series on ML fundamentals for software engineers, uses OpenAI's tiktoken as a concrete example to illustrate these constraints.