12% of LLM API Calls Returned No Answer, Exposing Hidden Costs in AI Pipelines

An analysis of 5,087 LLM API calls made between late June and early September 2026 found that 622 calls — about 12.2% — failed to return any answer, a rate the authors describe as normal for production environments. The largest failure category was 503 errors, meaning the upstream provider lacked capacity, while the second largest was 400 errors caused by malformed client requests. Critically, these two failure types require different handling: 503 errors are worth retrying, while 400 errors will never succeed regardless of how many retries are attempted. Timeouts and client-side cancellations proved uniquely costly, as tokens were already being generated when those calls stopped, and any retry would bill the full input prompt again from scratch. The authors recommend retrying only on 429 and 5xx status codes, and capping retry budgets by token count rather than by number of attempts to avoid runaway costs on large prompts.