Qwen 3.8 27B AI model inference speed increased over 100x on Nvidia hardware

Researchers significantly accelerated the Qwen 3.8 27B large language model's inference speed on NVIDIA B300 GPUs. The model's default single-stream performance was about 104 tokens per second on one GPU. By employing a custom stack with eight GPUs and 256 concurrent streams, the team achieved a throughput of 9,825 tokens per second. A separate deployment by Fireworks AI on the same eight-GPU setup reached 16,931 tokens per second. This performance gain required systematically overcoming five key bottlenecks, including memory latency and multi-GPU overhead.