Why AI test systems need a failure ledger, not just a pass/fail score

Software engineer Derek Wang argues, in an essay published on DEV Community, that AI harness testing should be modeled on philosopher Karl Popper's principle of falsification — advancing knowledge by eliminating wrong answers rather than accumulating right ones. Wang contends that a test system capable of recognizing and classifying failures is far more valuable than one that simply returns a pass or fail verdict. He describes a regression suite his team built, called fulltest, which maintains a structured ledger cataloguing each class of failure along with its root cause and known repair strategies. The system distinguishes between known failures — previously documented debts — and unknown failures, treating the latter as the most critical signal for genuine improvement. Wang concludes that fast, trustworthy feedback is what enables AI agents to make bold correct changes while remaining cautious about potentially harmful ones.