Stop Guessing API Timeouts: A 20-Minute Calibration Method for Model Servers

·1 views

Poorly configured timeouts on free model servers often stem from arbitrary guesses rather than measured data, causing either premature failures or silent stalls. A developer shared a calibration approach that involves sending 100 sequential requests to an endpoint and recording time-to-first-token (TTFT) and total duration metrics. Using Python and the httpx library in streaming mode, the method computes percentiles — p50, p90, p95, and p99 — to reveal the true latency distribution of a server. Sample data from the approach showed a stark 8x gap between median and 99th-percentile TTFT, illustrating why median-based timeouts can silently drop valid requests. The method takes roughly 20 minutes to run and is designed to replace instinct-based timeout settings with reproducible, data-driven thresholds.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

APIC Image Processing Tool Moves From Desktop App to Browser Platform

A developer has launched APIC Web, a browser-based version of APIC (Advanced Image Processing Center), eliminating the need to download or install any software. The platform allows users to open a website, sign in, and begin processing images immediately through a streamlined workflow. The web version was built to improve accessibility, making the tool practical for quick tasks, shared computers, and users who prefer not to install additional applications. APIC Web is now publicly available, and the developer is actively seeking user feedback to shape the platform's feature roadmap.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Builds Free macOS App to Manage 40+ AI-Assisted Side Projects

A full-stack developer with 15 years of experience found that adopting AI coding tools Cursor and Claude Code increased his project output roughly tenfold within six months, leaving him with over 40 active projects in various states of completion. The surge in productivity created a new bottleneck: locating and launching projects had become more time-consuming than building them. To solve this, he created Dev WorkDir, a free macOS menu bar application that centralises project management locally on the user's device. The app auto-detects 29 programming frameworks, supports 26 code editors, and allows users to open or run any project with a single click. Dev WorkDir is available on the Mac App Store at no cost and operates entirely offline with no cloud syncing or user tracking.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

58-Day LLM Agent Study Finds 43% of Failures Were Silent Output Shape Errors

A team running AI agents for real back-office work — including bookkeeping — logged 6,768 failures over 58 days and published the records as an open dataset. Contrary to expectations, 43% of failures were not hallucinations or refusals but 'shape errors,' where the model returned a correctly formatted HTTP 200 response with fluent text that downstream systems could not parse. The single most common failure, occurring 2,535 times, was a missing verdict line that a review agent's contract required, silently breaking the pipeline while human spot-checks showed nothing wrong. A secondary cluster of 1,048 failures involved language errors — including Chinese characters in Japanese output and mixed-language gibberish — often passing length and fluency checks undetected. The team recommends machine-checking output structure before any downstream stage reads it, logging failure reasons rather than just failure events, and measuring produced artifacts rather than agent invocations.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

EnvHarness Claims to Convert Static Data Into Live Agent Training Environments

A newly published research paper introduces EnvHarness, a method designed to transform static content — such as text, code, and game states — into interactive environments where AI agents can take actions and receive feedback. The core problem it targets is that most data used in agent training is inert, offering no response to an agent's actions. However, independent validation is still absent, as no external labs have yet tested agents trained in EnvHarness-generated environments against those trained in established, hand-built simulators. Critics note that auto-generating functional training environments from raw data still requires significant manual specification of actions, world responses, and reward signals, which undermines the cost-saving premise. EnvHarness is currently considered a promising but unproven approach, and its real-world value will depend on reproducible results from teams outside the original authors.

0 comments Read more at DEV Community