Client-side token bucket limits LLM API calls by both requests and tokens per minute
LLM providers often cap both requests per minute and tokens per minute, so a few large prompts can hit the token limit while many small ones hit the request limit first. Retrying after a 429 error wastes latency, so the article proposes a client-side approach instead. It uses two continuously refilling token buckets, one for requests and one for tokens, and makes each call wait until both have capacity. Token usage is estimated before sending and then corrected with the actual usage the API reports. The author includes a tested Node.js implementation, which uses a single queue to prevent starvation and rejects requests that could never fit the token limit.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in