How LLMs Actually Process Text: Tokenization Explained for Developers

Large language models never process raw words — instead, text is first broken into fragments called tokens and converted into sequences of numbers before the model sees anything. A token is typically a piece of a word rather than a whole word, and a separate component called a tokenizer handles this conversion before any neural network processing occurs. Modern models use subword tokenization algorithms such as BPE or WordPiece, which balance vocabulary size and sequence length by splitting rare words into smaller reusable pieces. This design explains several practical AI engineering realities, including why APIs charge per token, why long prompts cost more, and why context windows have hard limits. Developers can explore tokenization hands-on using OpenAI's open-source tiktoken library to see how different types of text — including code, emojis, and non-Latin scripts — map to token counts.