Fewer, Better-Ranked Chunks Cut RAG Costs and Improve LLM Answer Quality

A developer building a retrieval-augmented generation (RAG) chatbot found that setting top_k to 10 chunks per query was increasing costs without improving accuracy. Most of the extra retrieved chunks were tangential, forcing the language model to wade through noisy context and frequently miss the most relevant answer. The author introduced a reranking step to score candidates for relevance, then passed only the top 3 chunks to the model, reducing input-token costs on every query. Counterintuitively, answer quality improved when less context was sent, partly due to the 'lost in the middle' effect where models underweight information buried in long prompts. The key takeaway is that retrieval quality — not retrieval volume — determines both cost efficiency and response accuracy in RAG systems.