How to Properly Evaluate a RAG System Using Four Key Metrics

GenAI engineer Pranjul Rathour outlines a structured framework for evaluating Retrieval-Augmented Generation (RAG) systems, arguing that informal testing is insufficient for meaningful improvement. He recommends building a fixed evaluation set of 50 to 100 real-world questions, including 10 to 20 unanswerable ones to test whether the system correctly declines to respond. The framework tracks four core metrics: retrieval recall, faithfulness of answers to source documents, refusal accuracy, and overall answer correctness. Rathour warns against common pitfalls such as writing test questions that mirror document phrasing too closely, or using the same model to both generate and grade answers. He advises running this evaluation suite after every system change and tracking metric shifts over time to make informed, defensible development decisions.