Developer finds fuzzer silently skipped most tests during Python-to-Rust library port
A developer ported textdistance, a Python library with over 30 string-similarity algorithms, to Rust and documented the verification process in detail. Three independent checks were used to confirm behavioral equivalence: the original unmodified test suite, differential fuzzing on random inputs, and coverage comparison scripts. The most significant problem encountered was not in the ported algorithms themselves but in the differential fuzzing harness built to verify them. A shared constructor argument, as_set=True, caused 28 of the 37 algorithms to throw a TypeError on every test case, which the harness silently caught and skipped, falsely reporting zero divergences. The bug was only caught when the developer manually reviewed the harness code, and the incident is highlighted as a cautionary example of verification tools that appear to pass while testing very little.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in