Developer ports all five Gemma-4 models to AWS Inferentia2, details key engineering hurdles

A developer has successfully ported Google's entire Gemma-4 model family — E2B, E4B, 12B, 31B, and the 26B-A4B mixture-of-experts variant — to run on AWS Inferentia2 accelerators. Each model required a distinct compilation strategy, ranging from single-core tracing for the smallest model to NxD ModelBuilder with tensor parallelism up to TP=8 for the largest. All five models achieved token-for-token output matching their CPU fp32 reference, confirming correctness beyond surface fluency. Key challenges included unsupported architectures in existing vendor stacks, per-layer embeddings, cross-layer KV-sharing, and MoE routing that required custom workarounds. The writeup serves as a practical guide covering environment setup, softcap handling, attention overflow fixes, and MoE expert computation on Neuron hardware.