Zero baseline scores almost always signal a broken benchmark, not a real result

A developer on Reddit's RAG community shared a rare act of transparency by publishing the bugs in his own memory library benchmark rather than claiming a false victory. Two of his errors are widespread: one arm received nearly 10 times more context than another due to differing chunk sizes, making BM25 appear far superior until context budgets were matched. A competitor tool scored zero not because it failed, but because his harness silently truncated input data and ignored a misnamed API parameter without raising any error. These findings prompted a recommended minimum benchmark structure with three separate liveness gates — a positive control per arm, an evidence-in-context ceiling check, and a parameter-efficacy assertion. The core lesson is that a zero baseline score should be treated as a harness failure until all three gates explicitly confirm otherwise.