Do AI-Written Tests Actually Verify Anything? A Project Tackles the Question
A developer argues that tests generated alongside AI-written code can pass and turn CI green while verifying little about the implementation. The article illustrates this with a trivial always-true test and a discount fixture where a boundary bug at exactly 1,000 goes undetected by existing tests. To address this, the author built Veritas, an open-source project that examines which specification a test derives from and what it actually checks. Veritas uses mutation testing via mizchi/mutator to check whether tests detect code changes, rather than relying on the AI's own judgment. The author cautions that surviving or killed mutants do not by themselves prove correctness against the specification.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in