Load balancer dropped three wedged servers, but the auto scaling group kept counting them as healthy
A team running a tracking API on an EC2 auto scaling group found that three of their six instances had hung due to a deadlocked worker pool. The load balancer stopped routing traffic to them, but because the group used the default EC2 health check, it judged the machines healthy and never replaced them. Those idle wedged instances also dragged down the average CPU, so the group failed to scale out and pages timed out for most of a morning. The team switched the group to use load balancer health checks, added a grace period, and set an alarm on healthy host counts. They also narrowed the health endpoint so it no longer checks a shared database, which could otherwise trigger mass replacements.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in