HAProxy's 200ms LLM token delay explained: a kernel flag, not a buffer bug

·1 views

A benchmark comparing four reverse proxies showed HAProxy delivering LLM tokens 206 milliseconds late and in bursts, while nginx, Caddy, and Traefik passed them through in 2–3 milliseconds. An HAProxy engineer investigated and found the delay is not caused by application-level buffering but by the kernel's MSG_MORE flag, which HAProxy intentionally sets to hold small writes until more data arrives. This behavior stems from a documented default designed to batch small packets into full-sized ones, improving efficiency for the high-concurrency bulk HTTP traffic HAProxy typically handles. The delay disproportionately affects LLM token streams because their frames — around 60 bytes every 50 milliseconds — are too small to fill a packet, triggering the hold repeatedly. The behavior can be disabled with a single line of configuration, and larger frames reduce the delay significantly.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How Linux fork() Uses Copy-on-Write and Why It Can Exhaust Memory

When a Linux process calls fork(), the kernel does not immediately duplicate physical memory; instead, it copies the page table hierarchy and marks all entries as read-only. Any subsequent write by either the parent or child process triggers a hardware page fault, prompting the kernel to allocate a new physical page and copy the data. This Copy-on-Write mechanism keeps fork() fast under low-write conditions but becomes costly when workloads involve heavy, sustained writes to large memory regions. Systems running in-memory databases like Redis can experience write amplification, TLB thrashing, and uncontrolled memory growth during background snapshot operations. In extreme cases, the cascade of micro-allocations exhausts available RAM and activates the kernel's Out-of-Memory killer, terminating processes unexpectedly.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

NPU, DPU, QPU Explained: Which Specialist Chips Are Production-Ready Now

Hardware vendors are aggressively promoting three chip types — NPUs, DPUs, and QPUs — but their real-world utility varies significantly. Neural Processing Units (NPUs) are already proving their worth in production for low-power, on-device AI inference tasks like keyword detection and camera segmentation on mobile and edge devices. Data Processing Units (DPUs), such as NVIDIA BlueField and AMD Pensando, are similarly mature, helping large-scale infrastructure operators offload high-throughput networking, storage, and security tasks from general-purpose CPUs. Quantum Processing Units (QPUs), by contrast, remain largely confined to research settings and are not yet viable for mainstream production workloads. Engineers are advised to evaluate each chip against specific workload requirements rather than adopting them based on vendor marketing.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

EduVetta: New SaaS Tool Automates Repetitive Admin Tasks for Teachers

A developer has launched EduVetta, a SaaS platform designed to reduce the administrative workload teachers face outside of actual classroom instruction. The tool offers four core features: automatic lecture note summaries, AI-drafted question papers with answer keys, a simplified assignment submission system, and interactive quiz generation. Teachers can generate content from multiple input types including YouTube links, PDFs, or spoken requests, and students can access assignments without creating an account. Objective grading such as multiple-choice questions is automated, while subjective answers are intentionally left for human review to avoid errors that could harm students. The product is currently available with a free tier at eduvetta.com, though the developer notes that diagram- or equation-heavy subjects still require additional manual editing.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Structured Data Contracts Can Make Backlink Audits Easier to Debug

A technical design note proposes a small, structured data contract to improve how backlink audit tools record and interpret results. Rather than returning a simple pass-or-fail flag, the proposed format captures fetch state, document scope, link presence, and index or referral status separately. The distinction matters because a page can be reachable while a specific link is missing, or a link can exist without any evidence of indexing. The article also outlines best practices such as using URL parsers instead of substring matching, resolving relative links before comparison, and preserving prior successful observations during outages. The author notes the piece was prepared with AI assistance and is a design proposal, not a production benchmark or description of Google's internal systems.

0 comments Read more at DEV Community