Mutation Testing Tools Tested for File Cleanup After Forced Kills

A developer built a conformance suite to test whether four popular mutation testing frameworks — mutmut, cosmic-ray, Stryker, and PIT — leave source files in a mutated state when forcibly terminated mid-run. The test method hashed the source tree, launched each framework, waited until a file was visibly mutated on disk, killed the process, then hashed the tree again. Three of the four tools — mutmut, Stryker, and PIT — were found to be architecturally safe because they either mutate copies or work entirely in memory, never touching the original source files. cosmic-ray, however, edits the actual source file and restores it only after completing a mutation cycle, meaning a SIGTERM signal sent by a timeout before restoration leaves the file permanently mutated. The finding is described as narrow in scope, affecting only interrupted runs of cosmic-ray rather than its normal operation, but it highlights how architecture provides stronger safety guarantees than signal handling alone.