Google Gemma 4 E2B Runs Efficiently on Single TPU v6e Chip, QAT Variants Fail to Load

Google's 2-billion-parameter Gemma 4 E2B model was benchmarked on July 21, 2026, running on a single Google Cloud TPU v6e chip with 32 GB HBM in europe-west4-a. The plain gemma-4-E2B-it checkpoint delivered 213 tokens per second for a single user, scaled to around 2,200 output tokens per second under concurrent load, and handled vision queries in roughly 200 milliseconds. All three quantization-aware training (QAT) variants failed to load due to a loader bug tied to the model's KV-sharing architecture, which omits per-layer norms for layers 15–34 that the vLLM loader currently expects. A fix has been filed upstream as issue tpu-inference #3225, with the proposed solution being to skip instantiating K/V-side parameters for KV-shared layers. Until the patch lands, developers are advised to use the plain checkpoint, as 4-bit weight compression offers minimal memory benefit given that the KV cache already operates in fp8.