FixProof: A CLI That Checks Whether a Coding Agent's Fix Actually Works

The author argues that a passing test alone does not prove an agent's fix is correct, since it may not show the test failed before or would fail again without the change. They built FixProof, a stdlib-only Python skill and CLI that reproduces the bug, applies the fix, reruns checks, and temporarily reverts the fix to confirm the test fails without it. In a slugify example, the naive fix turned a double space into a double dash and was flagged as FAILED, while a corrected fix returned VERIFIED. FixProof reports statuses such as VERIFIED, FAILED, PARTIALLY VERIFIED, UNVERIFIED, and BLOCKED, and the author notes the agent still chooses the reproduction command. The tool is at version 0.3.0, released under the MIT license, and the author is seeking feedback.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in