Engineering Team Cuts RAG Pipeline Latency 40% Using Bayesian Search and Hybrid Retrieval

A development team rebuilt their retrieval-augmented generation (RAG) pipeline from the ground up after standard production deployments proved inadequate for legal, API, and support document types. The team replaced fixed 512-token chunking with document-specific strategies — including recursive, semantic, and agentic chunking — achieving recall@10 scores between 91% and 97% across content categories. They combined vector search with BM25 keyword search using Reciprocal Rank Fusion, then applied a cross-encoder reranker to narrow 50 candidates down to 5 final results, adding only 50ms while improving recall by 15%. A query transformation layer was also introduced to handle poorly phrased user inputs before retrieval begins. Together, these changes reduced end-to-end query latency by 40% compared to the original pipeline.