Dense Retrieval Outperforms Keyword Search Even on Identifier-Heavy Queries

A retrieval benchmark tested BM25 keyword search against dense vector search on 1,771 queries drawn from Unix and Android Stack Exchange subforums, sorting queries into lexical and semantic categories. Contrary to conventional wisdom, BM25 performed worse on queries containing identifiers like file paths and version numbers than on purely semantic ones. The root cause was that 54% of identifier-containing queries had no matching identifier in the relevant answer documents, leaving exact-match search with nothing to latch onto. Dense retrieval using all-MiniLM-L6-v2 outperformed BM25 by a larger margin on lexical queries than on semantic ones, with results confirmed across both subforums independently. The findings challenge the standard hybrid-search rationale that keyword search reliably handles keyword-style queries.