Hybrid Retrieval with BM25 and Reranker Outperforms Dense-Only Search on Technical Docs

A developer building a retrieval layer for a personal wiki found that dense-only vector search consistently failed to surface exact matches for technical identifiers like config keys, error strings, and CLI flags. On a 10-query evaluation set, dense-only retrieval placed the correct article in the top three results for just 5 queries, while a hybrid approach combining Qwen embeddings, BM25, and Reciprocal Rank Fusion (RRF) with a reranker improved that score to 8. Attempts to fix the problem by upgrading to a larger embedding model or using LLM-based query expansion yielded minimal gains while adding latency and non-determinism. Serving a cross-encoder reranker through Ollama produced silently incorrect relevance scores, because GGUF conversion strips the classification head the model relies on for ranking. The author concludes that technical corpora require lexical retrieval alongside semantic search, as embedding models are structurally unable to treat exact identifiers as discriminative tokens.