Why Cache Hit Rate Is the Key Metric for LLM Agentic Workloads

Prefix caching in large language model (LLM) serving reuses previously computed key-value caches when a request repeats the same prompt prefix, avoiding costly recomputation. This mechanism is especially impactful for agentic workloads, such as coding agents like Claude Code or Cursor, where every iteration re-sends the full accumulated context to the server. Without effective caching, time-to-first-token grows with each step as the model recomputes an increasingly long context, slowing down the entire agent loop. Caching also reduces costs significantly, since prefill compute scales with context length and agentic sessions can easily exceed 100,000 tokens. The cache hit rate metric — measuring how often requests are served from cache rather than recomputed — is therefore one of the most informative benchmarks when evaluating LLM serving infrastructure for multi-turn or agentic use cases.