Developer ports Gemma 4 to pure JAX, exposes key GPU-TPU compatibility limits

A developer has built a pure JAX port of Google's Gemma 4 language model and tested it across Cloud TPU v5e, v6e, and an NVIDIA T4G GPU on AWS, aiming to verify how truly portable JAX code is across accelerators. The project found that Gemma 4's irregular architecture — including two attention head dimensions and a complex KV-cache sharing scheme — caused failures in other frameworks like vLLM but was handled cleanly by JAX's XLA compiler. One critical portability issue discovered was silent performance degradation: using bfloat16 on pre-Ampere NVIDIA GPUs causes XLA to silently emulate it via fp32, severely hurting decode speed without any error or warning. To address this, the port now automatically detects GPU compute capability at runtime and selects float16 on older Turing GPUs and bfloat16 on TPUs and newer GPUs. A second limitation involves the fused W4A16 Pallas kernel, which is tiled for TPU memory and does not directly translate to GPU hardware, representing a genuine architectural difference rather than a software bug.