High test coverage hid nine real flaws in Engrava until mutation testing exposed them
Engrava's test suite boasted 3,845 tests and 94.22% line coverage, yet a mutation audit run before the 0.6.0 release uncovered nine distinct weaknesses across three categories. The core problem is that line coverage only confirms a line of code was executed, not that any test would detect if that line were incorrect or removed. Six of the findings involved validators that checked an input value but then continued using the original, unvalidated object — meaning the validation result was silently discarded. Those six issues were only reachable by code already running inside the same process, posed no cross-trust-boundary risk, and required no CVE or patch to prior releases. All nine issues were fixed, with the team noting that even unexploitable guards are worth correcting because the defensive pattern applies in contexts where callers are less trusted.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in