Why Fraud Detection Teams Should Never Have Abandoned BM25 for Vector Search

BM25, a keyword-based ranking algorithm from the 1990s, consistently outperforms modern vector embeddings when searching for exact identifiers such as decline codes, BIN numbers, and rule IDs commonly used in fraud analysis. A practical test on a sample e-commerce fraud queue showed that vector search misranked results for the query 'decline code 4863', surfacing a record with zero mentions of the code above records that contained it explicitly. BM25, by contrast, correctly ranked both matching records at the top due to its inverse document frequency weighting, which rewards rare, specific tokens. The core problem is that many engineering teams silently retire their keyword indexes when migrating to vector databases, without measuring the impact on identifier-based queries. Experts recommend running both systems in parallel and fusing their results using Reciprocal Rank Fusion, since the keyword index adds negligible storage cost while preventing missed matches that could allow fraud patterns to repeat undetected.