How LLM Context Windows Work and Why AI Forgets Mid-Conversation

Large language models rely on a fixed 'context window' — a working memory limit that determines how much text, code, or instructions the model can actively process at once. Unlike human long-term memory, LLMs re-read the entire conversation from scratch with every new message, meaning older content gets dropped once the limit is exceeded. When earlier context falls outside this window, the model can no longer reference it, which often leads to hallucinations, contradictions, or re-introduction of previously fixed errors. Context windows are measured in tokens — small text units produced by a tokenizer — where roughly 100 English words translate to about 150 tokens. Modern transformer-based LLMs also use contextual embeddings, which represent words as dynamic numerical vectors that shift meaning based on surrounding text, enabling more nuanced language understanding within the available window.