Silent CI: Misconfigured dependency caused 168 tests to skip unnoticed for two months
A developer discovered that 168 pytest tests had silently stopped running in CI for two months after a missing dependency declaration caused them to skip rather than fail. The root cause was use of pytest.importorskip(), which skips tests when a dependency is absent and exits with code 0, keeping the build green. Because the dependency was installed locally but not listed in the project's dev extras, CI quietly omitted those tests without any warning or error. The issue only surfaced when the developer compared test counts and noticed the skipped-test tally had jumped from 6 to 168 on a specific commit. The key takeaway is that a green CI build only confirms that the tests which ran did pass — it makes no assertion about how many tests actually ran.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in