RAG System Accuracy Jumped from 38% to 87% by Fixing Chunking and Retrieval

A retrieval-augmented generation (RAG) assistant built for fraud analysts was producing wrong answers, leading stakeholders to blame the underlying AI model and consider replacing it. Investigation revealed the real culprit was poor document chunking — splitting text every N characters mid-sentence — which destroyed the meaning of policy rules before they could be indexed. The fix involved restructuring chunks along natural document boundaries and adding overlaps to prevent rules from spanning chunk borders. A second improvement introduced hybrid retrieval, combining BM25 keyword search for specific identifiers like decline codes with dense embeddings for pattern-based queries, then merging results via reciprocal rank fusion. Measured against a golden evaluation set drawn from real analyst search logs, recall@5 improved from 38% to 87% without any changes to the language model itself.