nginx Buffering Delayed Claude Streaming Responses, Raising First-Token Time to Seconds

·19 views

A developer found that their Claude chat app streamed smoothly locally but stalled for several seconds once deployed behind nginx. The cause was nginx's default proxy_buffering, which collects upstream data and sends it in chunks, turning token-by-token server-sent events into delayed bursts. Replaying 200 prompts showed median time to first token rising from 0.71 seconds to 3.4 seconds, with some replies arriving as a single block. Disabling proxy_buffering on the streaming route, or sending an X-Accel-Buffering: no header from the app, resolved the issue, though a regex location rule initially overrode the fix. After the change, median first-token time dropped to 0.76 seconds.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

ShellGhost: A Local Open-Weight Assistant That Lives in Your Terminal

A developer has published a project called ShellGhost, a local open-weight AI tool designed to run inside a terminal. The announcement appeared on DEV Community under the original headline describing it as a spirit that haunts the author's shell. The source article's body text did not render properly, so details about its features, architecture, and performance could not be confirmed. The project is presented as a personal experiment rather than a commercial product. Further specifics would require reading the original post directly.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Local LLMs vs Cloud APIs: VRAM Limits, Cost Break-Even, and Routing Design

Engineers are advised against relying exclusively on either on-premises hardware or hosted provider APIs, and should instead decide which workloads suit each. The choice depends on three factors: available GPU memory, the token volume at which costs break even, and data isolation requirements. Model weights must fit in memory, and quantization can cut memory needs roughly fourfold, while mixture-of-experts designs reduce per-token compute. Based on the article's estimates, a roughly $2,500 workstation costs about $83 monthly, and pays off against frontier cloud pricing at around 8 million tokens per month. Local deployment offers physical data isolation, whereas hosted APIs rely on contractual safeguards, and on-premises setups must manage growing KV cache demands for long contexts.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Confirm Two Independent CLI Runs Match Before Freezing a Transcript Baseline

A DEV Community article argues that a repository's behavior should only be frozen as a golden baseline once two independent runs yield identical transcripts, since disagreement means there is no reliable baseline. It recommends testing at the process boundary, capturing exit codes, stdout, stderr, and a SHA-256 digest of the output file tree rather than function return values. The author describes using fixture directories, isolated temp copies, and a short, explicitly documented set of regex normalizers for nondeterministic fields like UUIDs and timestamps. To expose ordering nondeterminism, the process runs twice under different PYTHONHASHSEED values, and any divergence is fixed by sorting the data rather than pinning the seed. The article includes a roughly sixty-line dependency-free Python harness for recording and checking these golden files.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Workbook Guide for Turning Douyin Product Demos into English Ads

A cross-border e-commerce content editor shares a workbook for adapting their own Douyin product demos into English-language ads. The guide separates two tasks: removing the Douyin watermark and translating on-screen Chinese subtitles. The author's rule is to clean the visual layer first and translate the text layer second, to avoid smearing new English text during cleanup. A worked gadget example shows the watermark removed, the subtitle re-created in English as a separate track, and the video reframed to 4:5 for feed placement. The author also notes that apparel demos are stricter because burned-in text is larger and closer to where watermarks sit.

0 comments Read more at DEV Community