Engineering Team Cuts RAG Pipeline Latency 40% Using Bayesian Search and Hybrid Retrieval

A development team overhauled their Retrieval-Augmented Generation (RAG) pipeline after standard production deployments revealed critical failures with legal contracts, API docs, and customer support tickets. The team replaced fixed-token chunking with document-type-specific strategies, including recursive, semantic, and agentic chunking, achieving recall@10 scores between 91% and 97% across content types. They also adopted a hybrid retrieval approach combining vector search and BM25 keyword search via Reciprocal Rank Fusion, followed by a cross-encoder reranking step that improved recall by 15%. Query transformation was added to handle poorly phrased user inputs before retrieval begins. Together, these changes reduced end-to-end query latency by 40% compared to the original pipeline.