Judged passage labels, not clicks, should gate RAG retrieval releases
The author argues that a RAG retrieval evaluator for a documentation-monitoring alert system should rely on judged relevance labels rather than click proxies as its release gate. Recall@K is computed against a versioned set of question-to-passage labels, and builds fail when agreed slices regress, while clicks serve as a separate online signal. Averaging across all queries can hide complete misses on high-risk cases such as removed configuration constraints, so per-slice reporting by change type, template, language, and freshness is recommended. Labels should be defined at the same granularity the retriever returns, and the corpus should be frozen during evaluation. The approach does not by itself measure ranking quality, citation correctness, or answer faithfulness.
