Why AI-Generated Tests Passing Green Does Not Mean Your Code Actually Works

A common but flawed software development practice involves letting an AI model both write a feature and generate its own tests, then treating a passing result as proof the code works. Since the model authors both the implementation and the assertions, any misunderstanding it has will be consistently reflected in both files, making tests pass without validating correctness. A reliable evaluation gate requires a frozen oracle — test files and fixtures that the code generator cannot modify — kept in a separate, read-only directory. High test coverage also does not signal quality when the tests themselves were generated against the same logic they are supposed to verify. True verification demands an independent script with a logged exit code, human-owned fixtures, and a strict separation between what the AI can write and what it is judged against.