Engineer runs Google Gemma 4 on AWS G5g instance, overcomes three major technical hurdles

A developer has published a detailed field report on successfully serving Google's Gemma 4 E2B model on an AWS EC2 G5g instance, which pairs an ARM-based Graviton2 CPU with an NVIDIA T4G GPU. The setup is notably rare, as G5g is the only AWS instance to combine an NVIDIA GPU with a Graviton processor, meaning most mainstream AI software stacks lack support for its compute capability 7.5 on ARM64. Three key obstacles had to be overcome: the absence of sm_75 support in ARM64 vLLM container images, missing CUDA toolkit components in AWS's own Deep Learning AMI, and a Gemma 4 model architecture incompatibility with older versions of the Transformers library. The solution required building vLLM v0.27.2rc0 from source targeting sm_75, leveraging AWS's maintained ARM64 GPU DLAMI which ships PyTorch 2.12 with CUDA 13.2 and includes sm_75 support. After applying a single patch, the setup achieved 43.1 tokens per second in single-stream greedy decoding with a 329,579-token KV cache.