SGLang Outpaces vLLM on Prefix-Heavy Agent Workloads, but Gaps Shrink or Reverse Elsewhere

Benchmarks of multi-turn LLM agent workloads with an 8,000-token system context show SGLang's RadixAttention cutting time-to-first-token from 440ms to 38ms, while vLLM v0.6 with prefix caching reduced it from 480ms to 92ms. The gap stems from data structures: SGLang stores the KV cache in a radix tree that reuses any common prefix, whereas vLLM's block-based prefix caching is less effective when overlaps don't align to block boundaries. A separate 4,096-token test at 75% prefix overlap showed SGLang reaching a 78.6% cache hit rate versus 41.2% for vLLM, with substantially lower latency percentiles. However, the advantage disappears when workloads lack shared prefixes, where the two engines performed nearly identically on Llama 3.3 70B. On a single H100 running Qwen2.5-7B, vLLM delivered higher raw throughput, though SGLang retained a notable time-to-first-token advantage.