What Tokens Really Are and Why They Determine LLM Costs and Limits

In large language models, a token is not simply a word but a fragment of text determined by a trained vocabulary, meaning one word can equal multiple tokens and vice versa. Each model — including GPT, Gemini, and Claude — uses its own tokenizer, such as Byte Pair Encoding, with vocabularies ranging from roughly 50,000 to 100,000 text pieces. Technical terms, long numbers, emojis, and non-English text tend to consume significantly more tokens than common English words, making them costlier to process via APIs. Every LLM enforces a context window, a hard cap on the total tokens it can handle per call, which must accommodate the system prompt, conversation history, retrieved documents, and the user's message combined. Misunderstanding how tokenization works can lead to unexpected API costs and context-window overflows, especially in applications that pass large amounts of text to the model.