SShortSingh.
Back to feed

Free AI lanes are limited by queue wait and slot time, not token counts

0
·4 views

A deterministic local simulation shows that a two-slot free lane with forty summarization jobs can miss a batch deadline even while token counters read zero. Token meters only advance when work completes, so queueing delay goes unrecorded. The author argues that slot-seconds occupied is the more meaningful cost measure, and that retries consume concurrency slots as well as tokens. The drill uses declared parameters such as latency, jitter, and retry probability rather than measured provider data. It provides a standard-library Python script that writes a per-attempt ledger and summary metrics.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Fault-injection drill tests offline fallback in local-first apps by cutting the network mid-request

The article argues that a local-first app only proves its offline behavior when the network fails during an in-flight request, not between requests. It recommends a stdlib-only Python HTTP stub that can simulate four failure modes: connection reset, stalled body, truncated response, and normal success. Running the client against each mode is meant to expose bugs such as a truncated response being parsed and reported as success. The author stresses writing explicit assertions before testing, including a local-write latency budget, a check that queued items are not lost, and a check against duplicate sends on retry. Generating an idempotency key before the first attempt is presented as the safeguard against double-writes in reset scenarios.

0
ProgrammingDEV Community ·

Tabs in One: un manager Chrome gratuit care regrupează filele pe o singură coloană și eliberează RAM

Dezvoltatorul Tabs in One, o extensie gratuită pentru Chrome, a lansat un instrument care mută toate filele deschise pe o singură pagină cu o coloană lizibilă. Motivația este că bara orizontală de file devine greu de folosit când sunt deschise peste zece file, iar filele consumă multă memorie. Închiderea filelor originale după mutare eliberează imediat RAM-ul, iar utilizatorii pot restaura oricare sau toate filele cu un singur clic. Extensia oferă backup-uri automate, export și import în format JSON sau HTML și nu necesită cont sau abonament. Ea nu sincronizează sesiunile între dispozitive, iar codul sursă este deschis pentru verificare.

0
ProgrammingDEV Community ·

Developer builds SHADOW, a local platform for supervised AI agents on Windows

A computer science junior describes building SHADOW, a platform for deploying AI agents that run on the user's own PC with their own API keys. Each agent has its own chat, model, permissions, and schedule, and works from a goal by first producing a plan the user must approve. Agents can only access folders and run commands the user explicitly grants, must show risky commands before executing, and cannot control the screen without the user starting it. The author says the design grew out of frustration with manually coordinating separate chat, documentation, and terminal tools. SHADOW is free on the Microsoft Store for Windows 11 and requires no account.

0
ProgrammingDEV Community ·

Keeping a human approval gate between LLM agent proposals and system writes

An AI agent can prepare a consequential change without being permitted to carry it out, which lets organizations automate work while keeping a person accountable. The article uses a supplier onboarding example, where the agent validates data and drafts an ERP record but cannot create the supplier or release payments. It recommends a state machine with a persisted, versioned proposal that reviewers approve exactly, along with separate execution endpoints that accept only valid approval tokens. Approval records should capture identity, payload hash, decision and expiry, and executors need idempotency keys and reconciliation. The author suggests testing rejection, expiry, modification, duplicate and outage paths, and piloting one low-volume workflow with measured outcomes.