Engineering team cuts RAG pipeline latency 40% using Bayesian search and hybrid retrieval

A development team overhauled their retrieval-augmented generation (RAG) pipeline after standard production deployments — using fixed 512-token chunks and basic vector search — proved inadequate for legal, API, and support document types. The team adopted document-specific chunking strategies, including recursive, semantic, and agentic chunking, achieving recall@10 scores between 91% and 97% across content categories. They replaced pure vector search with a hybrid retrieval system combining vector search, BM25, and a cross-encoder reranker using Reciprocal Rank Fusion, which improved recall by 15%. A Bayesian optimisation layer was introduced to tune retrieval parameters, ultimately reducing end-to-end query latency by 40%. The rebuilt pipeline shifted the team's approach from ad-hoc semantic search to a measurable, tunable retrieval system suited for large-scale production use.