Why end-to-end accuracy can hide real failures in retrieval-augmented generation

A retrieval-augmented generation system can produce a correct-looking answer while retrieving the wrong passage, or misuse the right one by adding unsupported details. Scoring only whether the final answer matches a reference hides these failures and makes it unclear which component needs fixing. The article argues for evaluating the retriever and generator separately, since retrieval problems call for better chunking, embeddings, or ranking, while generation problems call for better grounding or abstention. It describes four possible outcomes for a single question, showing that only two count as success under end-to-end scoring. It then introduces retrieval metrics such as recall at k, precision at k, and mean reciprocal rank to check whether relevant evidence is present and ranked highly.