Engineering team cuts RAG pipeline latency 40% using Bayesian search and hybrid retrieval

A development team rebuilt their retrieval-augmented generation (RAG) pipeline from scratch after standard production deployments caused latency exceeding one second per query and poor recall accuracy. The team replaced fixed 512-token chunking with document-type-specific strategies, including recursive, semantic, and LLM-driven agentic chunking, achieving recall@10 scores between 91% and 97% across content types. They combined vector search and BM25 keyword search using Reciprocal Rank Fusion, then applied a cross-encoder reranker to narrow 50 candidates down to 5, adding only 50ms while improving recall by 15%. Query transformation techniques were also introduced to handle poorly phrased user inputs before retrieval. The resulting pipeline delivered a 40% reduction in end-to-end latency compared to the original demo-grade setup.