How AI Text Generation Works: One Token, One Probability Roll at a Time

Large language models generate text by repeatedly solving a single question: given everything written so far, what is the most probable next token? Rather than always selecting the highest-scoring word, the model performs a weighted random sample across its entire vocabulary, which is why the same prompt can produce different responses on different runs. Context heavily shapes these probabilities — adding a phrase like 'write this like a horror story' can shift a word's likelihood from near-zero to dominant. Developers can override this sampling behavior with 'greedy decoding' (always picking the top token) to get consistent, reproducible outputs, useful for structured tasks like JSON extraction. What appears to be AI memory or personalization is actually the model's prior conversation context being re-fed into its input window each time, reshaping its probability distribution rather than drawing on any stored knowledge of the user.