SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Groq Rate Limit Counts Declared Max Tokens, Not Actual Output, Tripping Developers

0
·3 views

Groq's free tier enforces an 8,000 tokens-per-minute (TPM) limit based on the max_tokens value declared in a request, not the tokens the model actually generates, meaning a 20-token prompt with max_tokens set to 8,192 will be rejected even though nothing was produced. A developer testing 14 Groq-hosted models found that four of them returned a 413 error solely due to the declared ceiling exceeding the TPM cap. The rate limit operates as a rolling window shared across all models, so a model that passes one request may fail an identical one minutes later as the shared budget depletes. This behavior poses a particular risk for AI agents, where tool schemas already consume significant prompt tokens and most frameworks default to high max_tokens values. Developers are advised to set max_tokens to only what is realistically needed — around 1,500 — to avoid requests being blocked before the model processes them.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Developer builds Electron desktop app that streams torrents before download completes

A developer has created CinePeer, a desktop video streaming application built with Electron and WebTorrent that allows playback to begin before a torrent finishes downloading. The app uses TMDB metadata to provide a browsable interface similar to Prime Video, with torrenting handled invisibly in the background. The core technical challenge was bridging HTTP range requests from an HTML5 video player to WebTorrent's streaming API, solved by spinning up a lightweight local HTTP server per torrent in Electron's main process. Because browsers restrict WebTorrent to WebRTC-only peers, the architecture splits responsibilities between a React-based renderer and a Node.js main process communicating over IPC. An additional hurdle was a CommonJS and ESM compatibility conflict between Electron and WebTorrent, resolved using lazy dynamic imports to also improve startup performance.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Developer Builds Auditable Portfolio Chatbot Using Open Knowledge Format and Multi-Agent Routing

A developer has built a portfolio chatbot that uses Open Knowledge Format (OKF), a vendor-neutral Markdown-and-YAML specification, as a governed knowledge layer to control what information an AI can share publicly. The system employs a multi-agent workflow in which agents interpret visitor questions, route them through an approved catalogue, and generate answers only from records that meet predefined publication rules. OKF structures knowledge as directories of Markdown files with YAML frontmatter, enabling both human-readable content and machine-queryable metadata including type, status, tags, and verification details. The developer extended the format with a custom field to flag which records are visible to chatbot visitors, separating the knowledge representation layer from the publication policy layer. The project was motivated by a broader question about AI governance: how to determine which information a system is permitted to use before it generates a response.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Groq's /models API Lists Non-Chat Models Alongside Chat Models With No Labels

A developer testing Groq's GET /models endpoint found that 5 of 14 listed models fail standard chat completion requests — not due to outages, but because they are speech-to-text, text-to-speech, or routing models mixed into the same flat list. None of the entries include a modality field to distinguish chat models from other types, leaving developers to identify them by name alone. Additional testing revealed that certain models, including guard models and allam-2-7b, silently enforce lower token limits than commonly used values like 8,192, with no indication of this in the API response. The compound router model returned a rate-limit error attributed to a different underlying model, exposing routing behavior only through error messages. Developers building model pickers are advised to probe each model individually and cache actual responses rather than relying on the catalogue.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Developer builds 3D globe world clock with real-time day/night shader

A developer created Timetate, a world clock and meeting planner built around a 3D globe that visually displays the real-time day/night terminator tracking the sun's position. The globe is constructed from three concentric spheres handling the Earth surface, cloud layer, and atmospheric glow respectively, using five WebP texture maps kept at 1024px for mobile performance. The sun's direction is calculated without an astronomy library by deriving solar declination and subsolar longitude from the current UTC time and day of year. The simplified solar model can be off by up to 16 minutes, which the developer argues is acceptable since the error is smaller than the visual softness of the terminator band itself. A key technical challenge involved correctly passing world-space versus view-space normals in the vertex shader to ensure accurate lighting across the globe.

0 comments Read more at DEV Community