Google Gemma-4 Models Successfully Ported to AWS Inferentia2 After Compiler Workarounds

A developer has published a field report detailing the successful porting of Google's Gemma-4 language models — in 2B, 4B, and 12B sizes — to AWS Inferentia2 hardware using Neuron SDK 2.23. The standard AWS vendor toolchain, including optimum-neuron and the Neuron vLLM backend, failed to support Gemma-4's architecture due to unsupported features like cross-layer KV-sharing and mixed attention head counts. The developer bypassed these limitations by directly tracing the Hugging Face model graph with custom tensor-parallel sharding logic instead of relying on the vendor's model builder. Achieved throughput rates were approximately 44 tokens per second for the 2B model on a single core, 33–39 tokens per second for the 4B model, and 15 tokens per second for the 12B model using tensor parallelism. Pre-compiled model artifacts and Docker images have been released publicly on Hugging Face and Docker Hub for community use.