How to Auto-Retry GitHub Actions CI Jobs Killed by AWS Spot Instance Reclaims
A developer running self-hosted ARM64 spot EC2 runners on GitHub Actions faced a recurring problem where AWS reclaiming spot instances mid-run produced failures indistinguishable from real test failures. Both CI shards died within 13 seconds of each other after reaching 75% completion, triggering a cascading frontend deploy abort. To address this, the developer built a workflow_run listener that fetches failed job logs and checks for known runner-shutdown markers before automatically retrying only the failed jobs. The solution includes safeguards to avoid re-running stale attempts, stop retrying after a third failure, and skip action when logs are unreadable, preventing false recoveries. The two-file setup requires no extra dependencies beyond the GitHub CLI already present on hosted runners.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in