SShortSingh.
Back to feed

How Redis Sliding Windows and Token Quotas Protect LLM API Spending

0
·67 views

Rate limiting LLM APIs differs fundamentally from conventional APIs because a single model call carries real monetary cost, and agent loops can fire hundreds of requests autonomously. Developers must track both request counts and token usage separately, enforcing whichever limit binds first, since a one-line prompt and a 200-page document summarisation both count as one request but differ vastly in cost. Tenant identifiers used in rate-limit keys must be sourced from authenticated sessions, never from the request body, to prevent callers from manipulating or exhausting other tenants' quotas. Fixed time windows expose a known boundary exploit where a full quota can be consumed twice in rapid succession, making Redis sorted-set sliding windows a more reliable alternative when executed atomically via Lua scripts. Because actual token costs are only known after a model responds, the recommended approach is to reserve an estimated token count before the call and reconcile the difference once the response returns.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingHacker News ·

Report Says Chrome's AI Features Consume Significant Bandwidth

A post on Perishable Press claims that AI features built into Google Chrome are using a substantial amount of network bandwidth. The article's author raises concerns about the data consumption these background features generate for users. The source material provides limited detail, and the post had received 14 points on Hacker News with no comments at the time it was indexed. Further technical measurements or confirmation from Google were not included in the available information.

0
ProgrammingHacker News ·

Onlook Launches Open-Source Desktop Tool for Designing Directly in Code

Daniel, founder of Onlook, a Y Combinator W25 company, has launched an open-source desktop app that places a design canvas over a real codebase. The tool is aimed at designers who now need to work in code, letting them explore many ideas at once. Each design variation appears as a live frame running the actual application with real data, and changes arrive as pull requests that can be reviewed or discarded. The team said it rebuilt the product as an agentic design tool after earlier web and Storybook-based attempts, and that the project has drawn over 25,000 supporters and 100 contributors.

0
ProgrammingDEV Community ·

Running a self-built Google scraper at scale: blocks, silent failures and hidden engineering costs

A test estimate puts a self-run Google scraper at roughly $2,450 to $5,150 a month for 100,000 searches, with engineering hours making up most of that cost. The same volume through four commercial SERP APIs is priced between $200 and $725 a month. In a November 2026 test, a default headless browser was redirected to Google's /sorry/ block page on every request, and some newer failures return HTTP 200 with empty or wrong data. Google has since added JavaScript verification pages and poisoned result pages, and it stopped honoring the num=100 parameter, multiplying the requests needed per keyword. The authors argue that outsourcing to a SERP API shifts the detection burden to the vendor rather than removing it.

0
ProgrammingDEV Community ·

SerpApi Ranks Fastest Among Five Google SERP APIs in 30-Day Test

A 30-day comparison from October 20 to November 18, 2026 tested five Google SERP APIs on the same 100 short keywords daily. SerpApi was the quickest, with a median response time of 2.1 seconds and a 90th percentile of 4.9 seconds. DataForSEO, SearchApi and SerpApi had success rates of 99.8% to 99.9%, while Oxylabs failed most often at 92.4%. Failures tended to cluster on a few bad days rather than occurring at a steady rate. The results cover only whether a response arrived without error, not whether its contents were accurate.