Why background jobs hitting rate-limited APIs fail: six common myths

Background job queues can generate work faster than a third-party API's rate limit allows, and the mismatch causes trouble. Adding more workers beyond the limit only produces extra 429 rejections and wasted retries, so concurrency should be sized against the downstream limit. Rate limits are usually enforced per API key or IP rather than per process, so limits must be coordinated across all workers, for example with a shared token bucket in Redis. Immediate retries after a 429 are almost certain to fail, so the article recommends honoring Retry-After headers or using exponential backoff with jitter, re-enqueuing delayed jobs instead of sleeping in workers. Rate-limit responses should also be treated differently from genuine errors, since a 429 means "not now" rather than a permanent failure.