llm-d v1.0 on Kubernetes: KV-Cache Routing, Disaggregation, and Production Readiness
The author evaluates llm-d, an inference gateway that routes LLM requests to backends based on cache locality rather than simple round-robin distribution. llm-d adds inference-aware scheduling around engines such as vLLM and SGLang, using an Endpoint Picker to select backends and coordinating separate prefill and decode stages in disaggregated serving. The review identifies three separate decisions: cache-aware routing, flow control, and prefill/decode disaggregation, and suggests starting with routing in front of an existing replicated vLLM deployment. The assessment is a source-backed architecture and deployment review, not a completed benchmark, as the author did not execute the installation or reproduce the cited performance runs. The article also recommends recording version provenance from the release-1.0 branch and includes a preparation script that clones the repository without installing anything into a cluster.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in