Long context windows versus RAG: when to send full documents and when to search

Developers weighing long-context models against retrieval-augmented generation can choose based on document size and how often questions repeat. Google's February 2024 Gemini 1.5 Pro announcement introduced a standard 128,000-token window and an experimental window of up to 1 million tokens, with Google claiming a 99% fact-finding rate in its needle-in-a-haystack test at that length. Sending a full document with every question still adds token costs, although implicit caching can discount repeated prefixes when requests share the same start. A 2023 study by Liu et al. found that models used information placed in the middle of long inputs less reliably, showing a U-shaped accuracy curve. RAG reduces prompt size by sending only retrieved passages, but its results depend on whether the search step finds the evidence each answer needs. The article recommends testing both approaches on one's own data, since the 2023 findings may not hold for newer models.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in