Engineering Team Cuts RAG Pipeline Latency 40% Using Bayesian Search and Hybrid Retrieval

A development team overhauled their Retrieval-Augmented Generation (RAG) pipeline after standard production deployments — using fixed 512-token chunks and basic vector search — failed to perform reliably across legal, API, and support document types. The team replaced their single-strategy chunking approach with document-specific methods, including recursive, semantic, and LLM-guided chunking, achieving recall@10 scores between 91% and 97% depending on content type. They also introduced a hybrid retrieval system combining vector search and BM25 keyword search, fused via Reciprocal Rank Fusion and re-ranked using a cross-encoder model. The cross-encoder reranking step, which narrows 50 candidates down to 5, added only 50ms of latency while improving recall by 15%. The combined optimizations reduced overall query latency by 40%, bringing the pipeline from over one second per query to a significantly faster and more tunable system.