Engineering Team Cuts RAG Pipeline Latency 40% Using Bayesian Search and Hybrid Retrieval

A development team overhauled their Retrieval-Augmented Generation (RAG) pipeline after standard production deployments revealed significant performance gaps across legal, API, and support document types. The team replaced fixed 512-token chunking with document-specific strategies — including recursive, semantic, and agentic chunking — tailored to content structure and achieving up to 97% recall@10. They combined vector search with BM25 keyword search using Reciprocal Rank Fusion, then applied a cross-encoder reranker to narrow 50 candidates down to 5, improving relevance correlation from 0.75 to 0.92. A query transformation layer was also added to handle poorly formed user queries before retrieval. The cumulative changes resulted in a 40% reduction in end-to-end latency and a measured recall@10 of 95% across the production pipeline.