Ghost database from cancelled CI job silently faked a passing migration test

A database migration at an unnamed engineering team appeared to pass all pipeline checks in July but failed immediately in staging because it relied on a Postgres extension that was never explicitly installed. The root cause was traced to long-lived CI runner virtual machines where Docker Compose reused containers from previous jobs, including one from a different branch that had been cancelled mid-run three days earlier. Because the teardown step only executed on success, the cancelled job left its database — complete with the installed extension — running on the shared host. An audit of the four machines uncovered 61 orphaned containers and 140 volumes, some dating back to February. The team resolved the issue by switching to ephemeral runners destroyed after each job, scoping Compose project names to the job ID, running teardown unconditionally, and adding a preflight check to ensure no pre-existing containers are present before a job starts.