Separate Retrieval and Generation Scores Make RAG Failures Easier to Diagnose

A single aggregate quality score for retrieval-augmented generation (RAG) systems can signal that something went wrong but cannot pinpoint whether the fault lies in retrieval or answer generation. Using distinct metrics for each stage — such as Recall@K, Precision@K, MRR, and nDCG for retrieval, and a faithfulness judge for generation — allows engineers to triage failures more precisely. A faithfulness judge assesses whether generated answers stay grounded in the supplied context, but it can approve a response that accurately summarizes irrelevant retrieved material, meaning both scores must be read together. Building this evaluation harness early in development is recommended, since the pipeline is more transparent before its architecture solidifies. Reliable scoring also depends on well-defined relevance labels and ground truth, which ideally are established during the same process that creates the documents and test questions.