How a Hidden Database Deletion in Terraform Plan Led to Safer Deployment Practices

A Terraform plan described as a routine tag update concealed a managed database replacement buried 210 lines deep, which a team narrowly avoided executing by accident. The destructive change originated from a subnet group attribute modified in a shared module three weeks earlier, never reviewed against this specific environment. The incident exposed how Terraform's verbose plan output makes it easy for humans to miss critical destroy or replace operations, especially under low-alert conditions. In response, the team introduced automated policy checks that block any destroy or replace of protected resources and require a separately labelled change with a second approver. They also restructured pipeline output to surface counts of additions, changes, and deletions at the top, ensuring dangerous operations are immediately visible rather than buried in noise.