Benchmarking Six Inference Engines for Gemma 4 E2B on a Single Cloud TPU v5e Chip

A developer compared six inference serving engines running Gemma 4 E2B on one Cloud TPU v5e chip, including vLLM, SGLang-JAX, a pure-JAX engine, and TorchTPU variants. Each engine ran alone on an identical flex-start v5litepod-1 instance using the same chat prompts, load sweeps, and 1,319 GSM8K questions. Answer quality was broadly similar across engines, with SGLang-JAX scoring slightly ahead of vLLM. For single requests, SGLang-JAX delivered the highest throughput at about 160 tokens per second, while TorchTPU trailed significantly. Under heavy load, patched SGLang-JAX reached 4,249 tokens per second at 64 requests, about 2.5 times vLLM's throughput, though vLLM remained the engine that works with Gemma 4 E2B as shipped.