Rate-limiting proxy turns LLM API 429 errors into queued 200 responses
A developer built UnlimitedLLM, a Python proxy that sits between a coding agent and an OpenAI-compatible Chat Completions endpoint to manage request rates. It separates concurrency limits from per-minute admission limits, since a simple semaphore would release slots too quickly and fire bursts at the provider. Requests are placed in a priority queue, with an X-Priority header letting interactive work jump ahead of background jobs, and waiting requests are promoted over time to prevent starvation. The queue has a hard cap, returning a local 503 with Retry-After when full. In tests across four providers, the proxy returned 340 of 340 requests as HTTP 200 with zero 429 errors, though the limits used were the author's own settings rather than advertised quotas.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.


Discussion (0)
Log in to join the discussion and vote.
Log in