Green Checkmarks in Software Testing Have Always Been Claims, Not Proof

A software grader and AI model evaluator argues that passing tests have never reliably confirmed code correctness, a problem the industry largely avoids acknowledging. Google reported in 2016 that around 16 percent of its tests showed flaky behavior, and common CI fixes like retry counts mask failures rather than resolve them. A 2014 University of Illinois study found the main causes of flaky tests are concurrency, async waits, and order dependency — deterministic bugs that only appear under specific timing conditions. Researchers at UCL and Sheffield formally defined the broader challenge in a 2015 IEEE paper called 'The Oracle Problem in Software Testing,' which has since been cited over a thousand times. Compounding the issue, a pattern called Assertion Roulette — multiple assertions with no identifying messages — was found in roughly 62 percent of JUnit test classes across eighteen open-source Java projects.