How a Misleading Kubernetes Readiness Probe Silently Routed Users to Broken Pods

A software engineer discovered that two of five Kubernetes pod replicas were returning HTTP 500 errors on real endpoints while their health checks continued reporting a healthy 200 OK status. The incident began on a Tuesday morning when the checkout API's error rate quietly climbed to around 18% over twenty minutes, with no alerts or pod failures visible on any dashboard. Because Kubernetes relies solely on the configured readiness probe to determine whether a pod should receive traffic, both faulty pods remained in active rotation and kept serving real customer requests. Bypassing the load balancer and querying each pod directly revealed the mismatch: the /healthz endpoint was functioning normally while the actual business logic was failing. The root cause pointed to a common readiness probe configuration that checks only a lightweight health path rather than verifying the pod's ability to handle real application traffic.