GitLab's 2017 Backup Failures Show Why Nightly Restore Tests Matter

·1 views

In January 2017, a GitLab.com engineer accidentally deleted a PostgreSQL data directory on the primary server, causing an outage of about 18 hours. The team discovered that none of its five backup and replication methods worked reliably, including a pg_dump job that silently produced near-empty files due to a client-server version mismatch. Roughly six hours of database changes were lost, and data was eventually recovered from a staging copy taken for load testing. GitLab's postmortem blamed the incident on no one being responsible for testing the restore procedure. The article proposes a nightly script that dumps, uploads, downloads, and restores the backup into a scratch container, comparing row counts before signaling success.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How HMAC Request Signing with Timestamps Hardens API Security

Bearer API keys alone let an intercepted request be replayed indefinitely, tampered with, or reused against other endpoints. The article proposes request signing, where each call carries an HMAC-SHA256 signature computed over the method, path, body hash, and a timestamp. Servers reject requests whose timestamps fall outside a 30-second window and verify signatures with constant-time comparison to prevent timing attacks. The author provides Python client and FastAPI server implementations and stresses that deterministic JSON serialization, such as sorted keys and fixed separators, is necessary for signatures to verify correctly. The approach mirrors techniques used by AWS SigV4, Stripe webhooks, and GitHub webhooks.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Three-Stage Filter Stops False Golf Cart Geofence Alerts Near Course Edges

Naive geofence alerts for golf carts fire repeatedly when GPS jitter near the green's edge crosses the boundary, training riders and supervisors to ignore them. The proposed fix stacks three filters: discard fixes with poor reported accuracy, treat a 5 m band around the zone edge as undecided, and require a new state to persist for 10 seconds. These are implemented as a small Python state machine with a companion test file. Each filter alone is insufficient, but together they change the alert only when the evidence is sustained.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer launches VibePost, an AI platform for social media content creation

An independent developer has built VibePost, an AI-powered platform aimed at simplifying social media content creation. The tool is designed to bring idea generation, post writing, editing, and publishing schedules into a single workspace instead of spreading them across multiple apps. The builder notes that creating content consistently is often more time-consuming than expected, which motivated the project. VibePost is an independently built product under JavaSpace and is still being refined, with its creator seeking feedback from developers and indie hackers.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Pippo, a locally run AI agent, introduces itself and its freelance experiments on DEV

Pippo, a self-described AI agent that runs entirely on a local Windows PC rather than in the cloud, has introduced itself on DEV Community. It writes and tests code in Python, C++, JavaScript/TypeScript and shell, and is experimenting with CAD work in Fusion 360 and Blender. The agent says it is attempting to earn income by bidding on on-chain marketplaces such as OpenTask and TaskMarket and delivering work in USDC, though its first payment was still pending when the post was written. It plans to document its engineering work, earnings and field observations on a public blog.

0 comments Read more at DEV Community