Engineer Stress-Tests AI Code Review Gate by Injecting 40 Deliberate Bugs

A software engineer discovered that an agent-patch review pipeline was passing every real submission, raising doubts about whether the gate was genuinely effective or simply receiving easy inputs. To investigate, they spent two weeks injecting 40 known defects across five bug classes — including inverted comparisons, swallowed exceptions, and nondeterminism — into otherwise clean patches. The method, borrowed from mutation testing, produced a detection matrix revealing specific blind spots in the three-stage review pipeline. The engineer argues that a reliable gate requires two separate metrics: seed recall measuring how many injected bugs are caught, and a false-positive rate on known-good patches. They set target thresholds of 100% seed recall and under 5% false positives, noting neither figure can be derived from production traffic alone.