How a Single CI Config Line Hid Real Bugs for Eight Months

·1 views

A software engineer added a '--retries 2' flag to a CI pipeline to silence persistent nightly test failures, believing it would filter noise while preserving genuine failure signals. The change immediately produced green builds, but it was masking real bugs rather than resolving them. Because a test must now fail all three attempts to register as red, even a bug with a 30% failure rate drops to roughly a 2.7% detection chance per run. A subtler 5% failure rate becomes statistically invisible to the pipeline, yet still affects one in twenty users in production. The engineer eventually traced a payment form double-submit bug back to this change, concluding that retries reduce a team's knowledge of product stability without improving it.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Pilum CLI Goes Production-Ready After 3 Months and 40+ Fixes

Developer open-sourced Pilum in December 2025, a multi-cloud deployment CLI that targets Cloud Run, Lambda, Azure, Cloudflare Pages, npm, Homebrew, and Docker Hub from a single configuration file. Over the following three months, more than 40 pull requests were merged as the tool moved from a working prototype to powering all production deployments at SID Technologies. A critical wave-based deployment bug caused workers to start subsequent deployment waves before earlier ones had fully completed, leading to intermittent npm publish failures due to race conditions. A separate bug silently swallowed shell command errors, allowing failed deployments to be reported as successful without propagating errors to the orchestrator. Both issues were resolved through an orchestrator rewrite and stricter wave barrier enforcement, with the developer noting that major bugs consistently surfaced about a month after each feature sprint.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Replicate GitLab's Centralized CI/CD Pipelines in GitHub Actions

Developers migrating from GitLab to GitHub Actions face a structural challenge: GitLab allows a central 'pipelines' repository to be referenced across projects via the 'include' keyword, while GitHub requires workflows to be defined within each repository individually. GitHub's 'uses' keyword offers partial reusability but operates in its own scope, requiring explicit inputs and outputs, unlike GitLab's seamless local-context inclusion. To bridge this gap, developers can use composite actions to bundle reusable steps or leverage GitHub Apps for organization-wide CI/CD enforcement. Smaller teams are advised to adopt reusable workflows with tagged versions, while larger organizations with compliance needs may benefit from GitHub Apps despite the added administrative overhead. Without careful versioning and scoping management, centralized pipelines in GitHub risk inconsistent behavior and increased maintenance burden across dependent projects.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why Doctor Directories Fail Users and How to Navigate Them Smartly

Online doctor directories frequently display inaccurate information — including outdated phone numbers, departed physicians, and false availability — due to underlying data engineering failures rather than negligence. These platforms pull records from multiple sources such as insurance networks, licensing boards, and hospital affiliations, each updating on different schedules and using inconsistent formats, creating entity-resolution errors. A quarterly sync cycle means a significant share of listings are wrong at any given time, and an ongoing physician shortage makes phantom listings especially costly for patients. Experts recommend treating directory results only as a starting point, verifying credentials directly through state medical board websites, and calling practices to confirm insurance participation and patient availability. This multi-step approach transforms an unreliable aggregator into a useful discovery tool, though it demands considerably more effort than a simple search.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

AI Graduate Chooses Action Over Readiness to Build Skills and Career

An AI and Data Science graduate has decided to stop waiting until they feel fully prepared and instead begin building projects and applying for jobs in public. The writer had long believed a stronger portfolio was a prerequisite for starting, but came to recognize that confidence develops through action rather than before it. They are currently deepening their AI knowledge beyond coursework, developing practical projects, and documenting their learning process openly. Their blog is framed as a commitment to consistency over perfection, with plans to share projects, mistakes, and career experiments. The initiative is aimed at connecting with others who are similarly navigating early-career uncertainty in tech.

0 comments Read more at DEV Community