How LLMs Really Work: Probability, Not Knowledge or Reasoning

Large language models (LLMs) are mathematical functions that predict the most probable next token based on patterns in training data, not systems that understand or reason like humans. When a model correctly states that Paris is France's capital, it does so because that token sequence appeared frequently in training data, not because it holds the fact as truth. This same mechanism explains hallucination: the model must always output a high-probability token, even when its training data offers no reliable answer. LLMs also have no awareness of events after their training cutoff and no memory between separate sessions, which is why techniques like retrieval-augmented generation and context re-injection exist. Understanding this token-prediction loop is foundational to grasping broader LLM behaviours, including why chain-of-thought prompting improves outputs and why agents can fall into repetitive loops.