How LLMs Turn Raw Scores Into Token Probabilities: The Softmax Explained

When a large language model processes a prompt, it first assigns a raw numerical score — called a logit — to every possible next token based on its learned weights. These logits can be positive or negative and do not naturally form a probability distribution. The Softmax function converts them into valid probabilities by exponentiating each logit and dividing by the sum of all exponentiated values, ensuring all outputs are positive and sum to one. This step amplifies differences between scores, making high-scoring tokens significantly more likely to be selected. A separate decoding strategy then uses this probability distribution to actually select the next token that the model outputs.