Engineering Team Cuts RAG Pipeline Latency 40% With Bayesian Search and Hybrid Retrieval

A development team overhauled their Retrieval-Augmented Generation (RAG) pipeline after standard fixed-token chunking and basic vector search proved inadequate in production environments. The team replaced one-size-fits-all 512-token chunking with document-type-specific strategies — including recursive, semantic, and agentic chunking — achieving recall@10 scores between 91% and 97% across legal, technical, and support content. They also adopted a hybrid retrieval approach combining vector search and BM25 keyword search, fused via Reciprocal Rank Fusion and re-ranked using a cross-encoder model, which added only 50ms of latency while improving recall by 15%. Query transformation techniques were introduced to handle poorly phrased user queries before retrieval. The combined optimizations reduced overall pipeline latency by 40%, bringing query response times down from over one second to more manageable levels at scale.