How a team discovered one-third of its deploys bypassed safety checks

A late-night deployment at a software team broke search functionality for 40 minutes after it skipped integration testing, staging, and mandatory second approval by using an emergency release path. An audit revealed that 31 of the previous 90 production deploys had used this fast-track route, none of which qualified as genuine emergencies. The root cause was a standard pipeline that took over an hour on average, making the six-minute emergency path a rational shortcut — and no system existed to track or flag its use. The team responded by labelling every deploy with its path, requiring a typed justification for emergency releases, and running full tests post-deploy with automatic rollback on failure. They also cut the standard pipeline from 52 minutes to 14 by parallelising tests and fixing flaky ones, reducing emergency path usage to just two out of 90 deploys.