How a Wrong rm -rf Command Wiped GitLab's Production Database in 2017

·5 views

On January 31, 2017, a GitLab engineer accidentally deleted the PostgreSQL data directory on the primary production server while intending to resync a lagging replica, destroying approximately 300 GB of data within seconds. When the team attempted recovery, all five backup and replication mechanisms in place were found to be non-functional or improperly configured, including nightly pg_dump jobs that had been failing silently due to a PostgreSQL version mismatch. GitLab.com remained offline for roughly 18 hours, with recovery relying on a manual snapshot taken six hours earlier for an unrelated test. The incident resulted in the permanent loss of around 5,000 projects, 5,000 comments, and 700 new user accounts, though Git repositories and wikis were unaffected. GitLab published a detailed postmortem on February 10, 2017, and CEO Sid Sijbrandij issued a public apology, making the incident one of the most widely cited database failure case studies in the industry.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Ghost database from cancelled CI job silently faked a passing migration test

A database migration at an unnamed engineering team appeared to pass all pipeline checks in July but failed immediately in staging because it relied on a Postgres extension that was never explicitly installed. The root cause was traced to long-lived CI runner virtual machines where Docker Compose reused containers from previous jobs, including one from a different branch that had been cancelled mid-run three days earlier. Because the teardown step only executed on success, the cancelled job left its database — complete with the installed extension — running on the shared host. An audit of the four machines uncovered 61 orphaned containers and 140 volumes, some dating back to February. The team resolved the issue by switching to ephemeral runners destroyed after each job, scoping Compose project names to the job ID, running teardown unconditionally, and adding a preflight check to ensure no pre-existing containers are present before a job starts.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Company finds 19 active shared mailboxes unmonitored, holding 1,400 unread emails

A customer complaint in April revealed that a shared mailbox printed on delivery notes had gone unmonitored for over a year after a mail migration broke its forwarding rule. An internal audit of 212 shared mailboxes found 64 unopened in 90 days, with 19 still receiving external emails and collectively holding around 1,400 unread messages, including carrier claims, customer cancellations, and council enquiries. The root cause was traced to a mailbox creation process that required no named owner or responsible manager. In response, the company overhauled its policy: every shared mailbox must now have an owning team and named manager, owners must confirm their mailboxes twice a year, and externally facing mailboxes are monitored so any message unread for five working days triggers a ticket. The original complainant, who had waited seven weeks for a response about a damaged pallet, was ultimately refunded.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Exposed .git Folders in Public S3 Buckets Can Leak Entire Source Code Histories

A vulnerability disclosed via HackerOne (report #2383486) showed that Mozilla accidentally uploaded its .git/ directory alongside marketing content to a public AWS S3 bucket. Because the deployment pipeline used git clone rather than a clean export, the entire repository metadata — including commit history, developer emails, and git remote URLs — was publicly accessible. Attackers can reconstruct the full repository, including secrets that were previously deleted from later commits, using just three shell commands. The root cause is a mismatch between developer intent and pipeline behavior: aws s3 sync uploads everything in the build directory, not just the intended build output. The fix is straightforward — either delete the .git/ folder before syncing or use git archive, which exports only tracked file content without repository metadata.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Finds 532 Stale URLs Two Weeks After GitHub Repo Transfer

A developer transferred an open-source mental health toolkit from an old GitHub account to a new named account, assuming the platform's automatic redirects would handle everything correctly. However, a routine SEO audit two weeks later revealed 532 hardcoded URLs across 47 HTML files still pointing to the old domain. The stale references affected Open Graph image tags, canonical URLs, and JSON-LD structured data, causing duplicate content signals for search engines and occasional broken preview cards on social media. A short Python script resolved all 532 instances in a single commit by replacing the old domain with the new one. The developer has since added a CI check to catch any recurrence and recommends using relative URLs and a centralized base URL config to avoid similar issues after future transfers.

0 comments Read more at DEV Community