Reader Tips Exposed Two Evaluation Harness Flaws, but Left Core Results Intact

A developer who had previously disclosed nine self-found bugs in their own code evaluation harness received two additional bug reports from readers, both targeting the harness's checks rather than its underlying data. One reader flagged that CPython's bytecode caching could allow stale .pyc files to execute original code instead of mutations, though the harness was already unintentionally protected by a line that excluded cache files during copying. A second reader suggested adding a deliberate negative control — a test case expected to score near zero — so that suspiciously high results would trigger investigation rather than go unnoticed. Both fixes were verified and implemented, yet the original results remained completely unchanged. The author notes that the episode highlights how code written for one reason can silently serve a more critical purpose, and how pre-registration alone cannot catch a broken instrument that produces the expected result.