Study Questions Claims That New KV-Cache Strategies Outperform LRU

·1 views

A technical discussion on Hacker News challenges the assertion that modern key-value cache replacement policies significantly outperform the classic Least Recently Used algorithm. The post, linked to a GitHub repository on agentic KV-cache strategies, suggests that LRU is more competitive than recent research papers imply. The thread has gathered 19 points and 9 comments from the community. The debate centers on whether newer caching approaches for AI inference workloads offer meaningful real-world gains over the well-established LRU baseline.

Read the full story at Hacker News

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Why Adding More Servers Can Make a Viral Traffic Spike Worse

When an app suddenly receives massive traffic, the instinct to add more servers can backfire if the root bottleneck is not identified first. In most cases, the database is the true chokepoint, not the application layer, which merely appears busy while waiting on slow database responses. Spinning up additional app servers only multiplies the number of connections hammering the same database, quickly exhausting its connection pool. Targeted fixes — such as indexing frequently queried data, caching hot keys, and queuing non-urgent tasks — address each bottleneck in sequence more effectively than scaling hardware. Engineers are advised to trace a single slow request end-to-end before making infrastructure changes, as each resolved bottleneck typically reveals the next one in line.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Prompt Caching Can Cost More If Cache Writes Never Get Read Back

A developer discovered that enabling prompt caching for an AI agent loop actually increased costs by about 25% instead of reducing them. The root cause was a volatile timestamp injected early in the system prompt, which changed the byte prefix on every request and prevented any cache hits. Prompt caching works as a prefix match over the fully rendered request, meaning any dynamic content placed before a cache breakpoint invalidates the entire cache entry. Cache writes cost 1.25x the base input price, while reads cost only 0.1x, so entries that are written but never read become a net surcharge. Developers can diagnose the issue by checking the usage fields: if cache_read_input_tokens stays at zero across repeated requests, something upstream is altering the prompt prefix.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Experiments With Delegating Full TDD Cycle to an AI Agent

A software developer who previously used AI agents only for production code decided to explore delegating the entire test-driven development (TDD) cycle — including test writing — to an AI agent. The shift was inspired by a presentation from Lada Kesseler on augmented coding, which demonstrated that an LLM could follow the red-green-refactor TDD flow while keeping the process visible and inspectable. Using a prompt shared by Kesseler as a foundation, the developer built a reusable TDD skill and deployed it via Codex CLI, with the agent pausing for approval at each step of the cycle. The developer later refined the workflow so the agent could complete a full TDD cycle before handing back control, reducing friction while still allowing frequent human intervention. The experiment highlighted how AI-assisted TDD can preserve process transparency without requiring the developer to retain direct control over every step.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Test-Driven Development Became One Developer's Answer to Coding Anxiety

Software developer and writer Edu describes how Test-Driven Development (TDD) became a personal coping mechanism for the anxiety and uncertainty he experienced throughout his programming career, dating back to his Computer Science studies in 2006. He recounts struggling with no reliable starting point for implementations, often needing to deliberately break working code just to trust that his manual tests were valid. After discovering Kent Beck's account of similar feelings, he began deliberately practising TDD in December 2022 while contributing to the open-source project now known as Seed4j. TDD's structured red-green-refactor cycle gave him a repeatable process to follow even on low-motivation days, independent of intuition or confidence. While he acknowledges broader technical benefits such as regression protection and simpler design, he identifies the psychological structure TDD provides as his primary motivation for adopting it.

0 comments Read more at DEV Community