Checkout 502 spikes traced to pods receiving ALB traffic after shutdown during deploys
During rolling deploys, the checkout service saw its 502 error rate briefly jump from about 0.02% to 4.8%, yet neither the app logs nor the target group health views showed any fault. Investigators found in the ALB access logs that failed requests mapped to pods still receiving traffic seconds after they were told to terminate. The checkout app had no SIGTERM handler, so pods exited within a second of the signal. The AWS Load Balancer Controller took four to seven seconds to deregister those pods from the target group. The 300-second deregistration delay on the target group made the gap worse by keeping the ALB routing to pods that had already gone.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in