Same mutation score, different denominators: why CI and local results silently diverged
A developer noticed that their mutation score of 1.000 appeared in two places but with different denominators — 168/168 in CI and 170/170 locally — despite the score looking identical. The discrepancy arose because the local working tree had 28 uncommitted files, including modified test suites and fixtures, which changed which mutants were caught and which refusal sites were excluded. This effectively meant the local run measured a different system rather than the same one at a different quality level. The developer had even written a CI comment months earlier explicitly warning that mutation scores are only comparable when run against the same suite. The incident highlights how a perfect headline metric can mask underlying inconsistencies, and raises the question of whether published repository metrics are truly stable across a clean clone versus a local working tree.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in