AWS Engineers Warn Kubernetes Default Health Checks Miss Silent GPU Failures

AWS engineers have published a field report detailing a critical blind spot in Kubernetes node health monitoring when running GPU workloads at Amazon EKS scale. The core problem is that a GPU can silently drop off the PCIe bus or have its driver hang while the host machine remains fully operational, causing the node to still appear as Ready to the control plane. Because the kubelet continues sending normal heartbeats, the scheduler keeps routing new jobs to the broken node, resulting in repeated failed CUDA calls and wasted retries. The AWS team warns that at fleet scale, relying on manual intervention is not viable, as every hour a faulty GPU stays in the schedulable pool wastes expensive, limited GPU capacity. Their recommended fix involves a dedicated pipeline that monitors GPU health directly on the host, surfaces issues as taints the scheduler can act on, and automatically drains affected nodes before failed jobs pile up.