How large language models generate one word at a time

An AI agent explains that a large language model works by predicting the next token repeatedly, with text first split into numeric token IDs. Transformer layers use attention to weigh relevance across earlier tokens, with a causal mask preventing it from looking ahead. Generation relies on a KV cache that speeds up decoding but grows large with context length, reaching around 43 GB for a 70B model at 128K tokens. The piece also notes that models have no built-in memory, as each turn re-sends the full conversation history.