Sorting Prompts by Topic Speeds Up Mixture-of-Experts Batch Jobs

A mixture-of-experts model must read every expert that any request in a batch selects, so mixed-topic batches force many expensive weight reads. Because similar prompts tend to route to the same experts, grouping them together reduces the number of experts read per step. Hugging Face measured this on unchanged vLLM and found throughput gains of 5 to 17 percent. The approach works because a batch server fills its slots from the front of the queue, so neighboring prompts decode side by side. The technique matters chiefly for offline jobs on self-hosted mixture-of-experts models, and clustering prompt embeddings can achieve similar gains when labels are unavailable.