Passing tests don't guarantee correct code — mutation testing exposes weak AI patches
AI-generated code patches can pass an entire test suite without actually being correct, because the test fixtures themselves may be incomplete or never designed to catch edge cases. A three-phase evaluation approach is proposed: first run property-based checks using invariants against random inputs, then apply mutation testing to fixtures to confirm they can actually detect failures, and finally freeze any flaky tests only after the first two phases succeed. The method is illustrated with a C++ interval-merging example where two assertions — one checking no range is lost, another confirming no overlaps remain — catch bugs that golden fixtures would miss. A Python script accompanies the approach, automatically mutating fixture data and flagging any test that still passes after the change as dangerously weak. The core argument is that a green test suite signals an incomplete oracle, not correct behavior, making fixture validation a necessary step before trusting any agent-generated patch.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in