How a developer built a production RAG system and measured what actually works

A software developer shared findings from building a production Retrieval-Augmented Generation (RAG) system over 62 ancient-history books, totalling around 46,000 text chunks. Using a 135-question test set, the standard tutorial setup — fixed 500-token chunks with top-5 retrieval — returned the correct passage only 35.2% of the time. Despite this low recall, the language model still generated fluent, confident-sounding answers even when the relevant text was never retrieved. The author warns that RAG failures are uniquely dangerous because fabricated responses are indistinguishable from accurate ones without prior knowledge of the answer. The piece argues that measurement against a fixed test set is the critical step most beginner tutorials omit, and that regular backend developers can build reliable RAG systems if they understand where the real pitfalls lie.