AI-Written Tests Can Pass Without Proving Anything — Here Is the Fix

When AI coding agents submit a patch, they often include tests in the same commit, creating a risk that the tests only validate themselves rather than the new behavior. A technique called the 'revert witness' addresses this by restoring production files to their pre-patch state and rerunning the test suite — if nothing fails, the tests are not actually pinning the change. Using a small tag-parser as an example, the article shows how a weak test suite stays green after reverting the patch, while a properly written suite fails on the reverted code, confirming it genuinely captures the new contract. A short Bash script, revert_witness.sh, automates this check by reading a configurable base commit and temporarily restoring original production files before running tests. The author argues this revert-witness step should be a mandatory part of both local staging and CI pipelines as AI-generated pull requests become increasingly common.