Why Your RAG System Gives Wrong Answers Even When All Metrics Look Fine

Many AI observability setups track model-level metrics like latency, token usage, and errors but fail to monitor the retrieval pipeline that feeds the model its information. A typical RAG system passes a user query through multiple stages — including query rewriting, search, extraction, deduplication, and reranking — each of which can silently introduce errors. Because standard LLM traces only capture the final prompt and response, they cannot reveal whether the model received outdated, duplicate, or simply wrong evidence. OpenTelemetry's generative AI conventions and tools like LangSmith are beginning to treat retrieval as a distinct, traceable component with its own metadata. Experts argue that production teams should measure retrieval-specific signals such as source freshness, citation coverage, and duplicate evidence to reliably diagnose why grounded answers go wrong.