More Context Doesn't Mean Better AI Answers — Here's Why It Backfires

Large language models advertise million-token context windows, but their effective working memory is far smaller, and performance degrades well before that limit is reached. This happens because attention mechanisms force every token to compete for a fixed share of the model's focus, so adding more documents dilutes rather than improves comprehension. Research by Liu et al. found that information placed in the middle of a long prompt is processed least accurately, making prompt position a critical but overlooked factor. Real-world data compounds the problem, as multiple similar or conflicting document versions make it harder for the model to identify the correct source. Practical mitigations include sending only the top three to five most relevant chunks, placing key instructions at the start and repeating the core question at the end of the prompt.