Terraform's sensitive marking hid a database change, causing nine minutes of failed writes
A pull request merged in March pointed an order service at a reporting replica, causing every write to fail for nine minutes with a read-only transaction error. The change was hidden in a Terraform plan because a secret in the ECS task definition marked the entire JSON-encoded container definitions as sensitive. Reviewers approved the plan line they usually approve, not realizing the database host change was masked. The team reverted and reapplied, then moved the secret into AWS Secrets Manager so the plan shows full diffs again. They also added a policy check that fails pipelines on fully masked production attributes and a startup check that refuses to report ready if the database is not a primary.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in