Developer finds hybrid RAG search underperformed vector-only retrieval in controlled test

A software developer built a retrieval-augmented generation (RAG) support assistant for a fictional B2B SaaS platform called Helix, grounding answers in a 100-document knowledge base of product docs, runbooks, and support tickets. The system passed production-readiness thresholds with a faithfulness score of 0.939 and context precision of 0.775 on a 50-query evaluation set. The developer hypothesised that combining BM25 keyword search with vector search via Reciprocal Rank Fusion would outperform vector-only retrieval, expecting the hybrid approach to handle both exact-term and semantic queries better. Controlled testing across all 50 queries showed the opposite: vector-only retrieval achieved a perfect Hit@5 rate of 100%, while hybrid search dropped to 94%, with lower document precision as well. The developer attributed the result to the corpus being mostly well-formed prose rather than text dense with exact-match triggers like error codes, where BM25 typically provides the most benefit.