Study finds only 1 of 50 company CI setups has a defensible test-gating defect
The author audited 50 well-starred enterprise repositories with CI pipelines in AI, DevTools and fintech to see whether test failures could be silently ignored. Only one repository, livekit/agents, currently has a defect that can be defended as a real problem, a 2% rate. The defect is a test step ending in '|| true' piped through tee, which masks pytest failures regardless of exit status. The author also notes that an earlier ranking of CI pain was contaminated by counting fork-branch and action_required runs as failures, so it was discarded. The first classifier also missed gates expressed through needs: dependencies, and it was expanded to cover more test runners, which shifted the count by one repository.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in