Four Common RAG Retrieval Failures and How Logging Each One Fixes Them

A developer building Retrieval-Augmented Generation (RAG) pipelines found that most errors blamed on the language model were actually caused by flawed retrieval, a realization that only became clear after adding detailed logging. Four distinct failure types emerged: missing answers in the knowledge base, semantically similar but contextually wrong chunks, model hallucination due to weak system prompts, and malformed chunks produced by token-count-based splitting. Hybrid search combining vector and BM25 retrieval proved more effective than switching embedding models for resolving keyword-sensitive mismatches, while cross-encoders offered a complementary reranking approach. Strict system prompts requiring the model to cite source passages and admit ignorance addressed cases where retrieval was correct but the model still extrapolated. The author concludes that logging chunk scores and sizes from the start is more valuable than any model swap, as most RAG problems originate in the retrieval and chunking stages rather than the LLM itself.