How LLMs Use Tokens to Process and Generate Text

Large language models do not process raw text directly; instead, input is first broken into smaller units called tokens before any computation begins. A token can represent a whole word, part of a word, punctuation, whitespace, or symbols, and its exact definition varies by model and tokenizer. Each token is mapped to a unique integer ID, which is then converted into a dense numerical vector called an embedding before being fed into the transformer. During text generation, the model predicts the next token one step at a time, appending each result to the sequence in a repeating autoregressive loop. Tokenization affects key practical factors such as API pricing, context window limits, memory usage, and inference latency.