Deleting replicas from a manifest cut checkout to one pod and caused a 40% error spike
A team had set replicas to 10 in its checkout Deployment manifest for two years, then added an autoscaler ahead of Black Friday. Following Kubernetes guidance, they removed the replicas line from the manifest so the autoscaler could manage the count. Because kubectl apply treats a deleted field as an instruction to restore its default, the Deployment dropped to one pod shortly after the change merged. Replacement pods took about ninety seconds to become ready, leaving one pod to handle lunchtime traffic for nearly three minutes, with errors peaking at forty percent of requests. The documentation had warned of this and recommended clearing the field from the stored last-applied annotation first. The team now runs kubectl diff on every pull request and requires a second approver for any change that lowers a Deployment's replica count.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in