Developer ports all five Gemma-4 models to AWS Inferentia2, shares key lessons

A developer successfully ported Google's entire Gemma-4 model family — E2B, E4B, 12B, 31B, and the 26B-A4B mixture-of-experts variant — to run on AWS Inferentia2 accelerators. Each model required a different hardware configuration, ranging from a single inf2.xlarge core for the smallest model to inf2.24xlarge instances with eight-way tensor parallelism for the largest. All five models passed token-for-token correctness checks against a CPU fp32 reference, confirming output accuracy across the family. Key technical challenges included handling per-layer embeddings and cross-layer KV-sharing in the smaller models, managing scale-related memory issues in the 31B dense model, and tracing all 128 MoE experts statically for the sparse 26B-A4B variant. The developer also documented shared environment setup steps and common pitfalls, such as a PATH misconfiguration that causes a libneuronpjrt import error.