Gemma-4 12B Ported to AWS Inferentia2 With Three Key Compiler Fixes

A developer has successfully ported Google's Gemma-4 12B instruction-tuned model to AWS Inferentia2 hardware, specifically an inf2.8xlarge instance with tensor parallelism set to 2. The model, classified as Gemma4UnifiedForConditionalGeneration, is an encoder-free multimodal architecture requiring the language model to be extracted from its unified wrapper before compilation. Three non-obvious issues had to be resolved: global attention layers with a single KV head needed correct query-head sharding, the logit softcap operation was safely dropped on-device since it does not affect greedy decode argmax, and a sliding-window attention buffer overflow required forcing eager attention instead of fused kernels. The resulting compiled model achieves token-for-token identical output to a CPU fp32 reference, with a first-token prefill latency of roughly 101 milliseconds and a memory footprint of approximately 12 GB per rank in bf16.