Engineering team cuts RAG pipeline latency 40% using Bayesian search and hybrid retrieval

A development team overhauled their retrieval-augmented generation (RAG) pipeline after standard production deployments proved inadequate for legal, API, and support-ticket documents. The team replaced fixed 512-token chunking with document-type-specific strategies, including recursive, semantic, and LLM-guided chunking, achieving recall@10 scores between 91% and 97% across content types. They combined vector search with BM25 keyword search using Reciprocal Rank Fusion, then added a cross-encoder reranking stage that improved recall by 15% at a cost of just 50 milliseconds. A query transformation layer was also introduced to handle poorly phrased user queries before retrieval. Together, these changes reduced end-to-end query latency by 40% compared to the original pipeline.