Engineering Team Cuts RAG Pipeline Latency 40% Using Bayesian Search and Hybrid Retrieval

A development team overhauled their Retrieval-Augmented Generation (RAG) pipeline after standard production deployments revealed critical failures with legal contracts, API docs, and customer support tickets. The team replaced fixed 512-token chunking with document-specific strategies — including recursive, semantic, and agentic chunking — achieving recall@10 scores between 91% and 97% across content types. They combined vector search, BM25 keyword search, and cross-encoder reranking using Reciprocal Rank Fusion, reducing the candidate pool from 50 to 5 results while gaining 15% additional recall. A query transformation layer was also added to handle poorly formed user queries before retrieval. The combined changes resulted in a 40% reduction in end-to-end query latency compared to the original pipeline.