Engineering Team Cuts RAG Pipeline Latency 40% Using Bayesian Search and Hybrid Retrieval

A development team overhauled their Retrieval-Augmented Generation (RAG) pipeline after standard production deployments revealed significant performance gaps with legal, API, and support ticket documents. The team replaced fixed 512-token chunking with document-type-specific strategies, including recursive, semantic, and agentic chunking, achieving recall@10 scores between 91% and 97% across content types. They combined vector search with BM25 keyword search using Reciprocal Rank Fusion, then applied a cross-encoder reranker to narrow 50 candidates down to 5 final results, adding only 50ms of latency while boosting recall by 15%. A Bayesian-informed search tuning approach was also integrated alongside query transformation to compensate for poorly worded user queries. The cumulative changes reduced overall query latency by 40% compared to the baseline pipeline.