Rust Engine Routes Some LLM Requests to CPU, Cutting Burst Completion Time 9.8%

A developer built mini-vllm-rs, a single-node LLM inference engine in Rust on Candle for macOS that can run CPU and Metal GPU backends concurrently. Initial routing tests favored sending most requests to the GPU, prompting a follow-up on whether selective CPU offload could help. Routing one of eight requests to CPU reduced a burst's completion time by 9.8%, and offloading six of 48 requests raised aggregate output throughput by 12.4%. The experiments used a quantized Qwen2.5 0.5B model on an Apple M2, where the GPU was roughly 35 times faster at prefill and 6.5 times faster at decode. The router scored backends by estimated pending and incoming prefill and decode work, using warm prefix caches on both sides.