Developer builds Rust LLM inference engine for macOS, explores heterogeneous serving trade-offs

A developer with a background in ML accelerators and OS virtualization built a single-node LLM inference engine called mini-vllm-rs entirely in Rust for macOS. The engine supports features like continuous batching and paged KV caching, and it allows CPU and Metal GPU workers to run concurrently. The project documented challenges such as performance drops when switching matrix operation kernels during batching, requiring targeted optimizations. The developer's analysis explores when splitting inference tasks between different hardware types, like GPU prefill with CPU decode, can justify the communication overhead. This work examines trade-offs in specialized hardware configurations currently being explored by major industry players.