Article URL: https://householderkarl.com/a-sick-people/ Comments URL: https://news.ycombinator.com/item?id=49182847 Points: 9 # Comments: 1
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Large language model inference involves two distinct workloads: a compute-heavy prefill phase that processes the entire prompt, and a latency-sensitive decode phase that generates tokens one at a time. When both phases run on the same hardware, a long-context prefill can stall all active decode streams on that engine, a problem known as head-of-line blocking. Prefill/decode disaggregation addresses this by routing each phase to dedicated server pools, preventing prefills from ever queuing ahead of decodes. A Go-based simulation demonstrated that splitting these pools reduced p99 inter-token latency from 88ms to 30ms, a 66% improvement, at the cost of a modest increase in time-to-first-token. This architectural pattern is already adopted in production serving systems such as DistServe, Splitwise, and vLLM with Mooncake.
A class of hard-to-catch software bugs in AI agents only surfaces when faults occur in a specific sequence — for example, a retry firing after a side effect causes a customer to be charged twice. Deterministic simulation testing (DST) addresses this by routing all sources of nondeterminism, such as faults, timing, and randomness, through a single seed, making any failure perfectly reproducible. The technique, used by systems like FoundationDB and TigerBeetle, also supports shrinking, which strips a complex failing scenario down to its minimal root cause. A Python demonstration showed that standard happy-path tests missed a double-charge bug, while seeded fuzzing caught it, replayed it identically, and reduced a four-fault sequence to the single fault responsible. DST is gaining broader attention as a practical method for testing agents that operate in unpredictable, fault-prone environments.
AI agents follow a strictly serial loop of reasoning, calling a tool, waiting for results, then reasoning again — leaving GPUs idle for up to 61% of wall-clock time in tool-heavy workloads. A technique called speculative tool execution addresses this by predicting the next tool call and running it in parallel while the model is still reasoning. If the prediction matches the model's actual call, the result is already available and latency is eliminated; if not, the speculative result is discarded and the correct tool runs instead. A Go-based demo using a learned pattern predictor achieved a 58% hit rate, reducing wall-clock time by 1.3x with no impact on output correctness. Four research papers published in 2026 — including Speculative Actions, SPORK, PASTE, and Speculate While You Reason — report real-world speedups of 20 to 48% using similar approaches.
Researchers at Letta (Lin et al., 2025) have proposed a technique called sleep-time compute, which shifts AI inference work to idle periods between user sessions rather than processing everything on demand. A background worker pre-answers queries likely to be asked again and compresses standing context into dense summaries, so the system can serve warm, instant responses when users return. Each pre-computed answer is tagged with the source version it was derived from, and a freshness check ensures stale answers are discarded and recomputed rather than served to users. In a demonstration across 400 queries — 70% predictable and 30% novel — foreground latency dropped by 57% and foreground cost fell by over half, while novel queries were still handled live. The approach is particularly suited to workflows where users repeatedly query the same documents or codebases, since the underlying context changes infrequently between sessions.
Discussion (0)
Log in to join the discussion and vote.
Log in