Sluice: Open-Source Proxy Adds Tenant-Aware Traffic Control for Self-Hosted LLMs

A developer has released Sluice, an open-source proxy layer designed to enforce quality-of-service policies for self-hosted large language model inference using vLLM. The tool addresses a gap in existing infrastructure: when GPU KV-cache capacity is under pressure, standard schedulers treat all requests equally regardless of tenant priority or service agreements. Sluice assigns tenants to one of three tiers — Guaranteed, Standard, or Best-Effort — and makes per-request admission decisions based on live signals including GPU cache usage, queue depth, and SLA violation rates pulled from Prometheus and PostgreSQL. Unlike Kubernetes or generic API gateways such as Envoy or Kong, Sluice can dynamically rewrite parameters like max_tokens or shed lower-priority traffic before requests ever reach the inference engine. The project is available on GitHub and is positioned as a complement to, not a replacement for, existing routing and scheduling infrastructure.