LLM Context Windows Degrade Well Before Their Advertised Limits

Large language models advertise maximum context windows — such as 200K tokens — but research shows measurable quality degradation begins around 130K tokens, well before that ceiling. A known issue called 'lost in the middle' means models pay far more attention to content at the start and end of a prompt, often missing instructions buried in the middle. The context window also covers combined input and output tokens, meaning long system prompts and chat histories can leave little room for the model's response, causing truncation without warning. Attention computation scales quadratically with input length, so longer prompts significantly increase both latency and API costs on every request. Developers are advised to count tokens across all prompt components before each request to catch bloat early and avoid silent quality loss on long inputs.