Slow Report Requests Caused Nginx to Mark Healthy Web Servers Down

A customer portal running four application servers behind nginx saw failures when branch managers ran a month-end statement report on the first working day of November. The report took longer than nginx's sixty-second read timeout, so nginx counted each timeout as a server failure and retried the request on the next server. Each request thus marked servers unavailable in turn, and at times all four were out of rotation, causing about one in six requests to fail with 502 errors. The servers themselves stayed healthy at around forty percent CPU, and the problem came from nginx's default failure settings. The report now runs in the background and is emailed, and it has its own upstream configuration that does not count timeouts as failures.