Study finds AI coding agents rarely write hollow tests, and flags those that cannot fail

The author built hollowtest, a tool that scans test files for tests that would still pass even if the code under test were broken. Examples include assertions with no matcher call, self-comparisons, unawaited promises, and checks hidden inside catch blocks. Running it on 18 well-known open-source repositories surfaced a few such tests in suites such as zod, swr, and requests. To test whether AI agents would produce hollow tests, the author ran 64 Claude Code sessions on eight small modules with planted bugs. The agents found 31 of 32 bugs when writing tests and refused to weaken tests under pressure, so the hook made no measurable difference.