Developer finds flawed RAG evaluation metrics led to wrong system decisions

A developer building a retrieval-augmented generation (RAG) CLI tool discovered that the metrics used to evaluate its performance were fundamentally broken, producing misleading results for over a week. The substring-matching method used to score retrieval hits incorrectly marked correct answers as misses due to minor formatting differences, such as '60,000' versus '60000', causing the measured hit-rate to read 13 percentage points lower than the actual performance. Fixing the scorer to use token overlap instead of exact substring matching revealed the system was performing significantly better than believed, and also changed conclusions about how many chunks to retrieve. A second flaw emerged when the developer found that evaluating on just 20 questions produced wildly unstable results — a 50% hit-rate at that sample size shifted to 65% with 100 questions using identical code and settings. The findings highlight how small errors in evaluation design can not only misreport accuracy but actively steer engineering decisions in the wrong direction.