Engineer Spent 3 Months Fixing Rankings When the Real RAG Problem Was Document Selection

A software engineer spent three months repeatedly swapping out ranking algorithms — including BM25, hybrid retrieval, and cross-encoder rerankers — in an attempt to improve a retrieval-augmented generation (RAG) system. Despite each change reshuffling search results, the correct architecture document never appeared in the top results because it was never entering the candidate set to begin with. The engineer eventually realized that ranking can only reorder documents already selected, meaning no algorithm could surface a document excluded at the selection stage. This led to a reframing of retrieval as three distinct layers: selection, assembly, and ranking — with selection being the most consequential yet most neglected. The engineer now argues that optimizing ranking while ignoring upstream selection is a fundamental misdiagnosis of retrieval quality problems.