LiteLLM PR fixes stale rate-limit counter that blocked traffic after its window expired
A pull request to the LiteLLM project addresses a bug in its parallel request limiter, where a local mirror of rate-limit counters kept reporting OVER_LIMIT after the time window had already rolled over. The stale state stemmed from a TTL anchored to the write time rather than the window size, meaning valid-looking counts could be attributed to the wrong window. The patch makes the local read path treat counters from expired windows as absent, applied across the pre-check and both fallback paths, without resetting state itself. The accompanying test checks that a stale counter no longer blocks after the window rolls and that the window correctly admits a new unit. At the time of the report the PR was still open, unreviewed by humans, and its mergeability status had changed to UNKNOWN.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in