Engineering team cuts RAG pipeline latency 40% with hybrid search and Bayesian tuning

A development team rebuilt their retrieval-augmented generation (RAG) pipeline from scratch after standard production setups caused latency exceeding one second per query and poor recall across document types. The team replaced fixed 512-token chunking with document-specific strategies — including recursive, semantic, and agentic chunking — achieving recall@10 scores between 91% and 97% depending on content type. They combined vector search with BM25 using Reciprocal Rank Fusion, then applied a cross-encoder reranker to narrow 50 candidates down to 5, adding only 50ms while boosting recall by 15%. Query transformation was also introduced to handle poorly worded user inputs before retrieval. The resulting pipeline achieved 95% recall@10 overall and reduced end-to-end latency by 40% compared to the baseline approach.