Why RAG Systems Need Independent Layer Metrics, Not Just Output Reviews

Retrieval-Augmented Generation (RAG) systems can produce fluent, confident-sounding answers even when the underlying retrieval is failing, making eyeball evaluation dangerously misleading. A real-world demo incident revealed a system retrieving the correct document only 30% of the time, yet appearing flawless because demo questions were hand-picked. Experts argue RAG evaluation must measure the retrieval and generation layers independently using metrics such as Hit Rate, MRR, Precision@k, and NDCG. Hit Rate at k=5 is considered the single most critical metric, with scores below 80% pointing squarely to a broken retriever. Without structured, labelled evaluation sets and layer-specific scoring, teams have no reliable way to identify where quality is being lost or where to focus fixes.