Single TPU Chip Tested for Small Agent Workloads Using Raw JAX After vLLM Gaps Found

A developer tested Google's Cloud TPU v6e-1 chip to serve small agent workloads of two to eight concurrent agents, finding that standard inference benchmarks optimized for high concurrency poorly represent real-world agent use cases. The experiment highlighted that serial agent loops with human users at the end are latency-bound, while machine-to-machine parallel fleets are throughput-bound, with 'goodput' proposed as a metric covering both scenarios. Context growth was identified as a key challenge, as decode speed on similar TPU hardware drops sharply from roughly 8,300 to 2,900 tokens per second as conversation history accumulates over many turns. Attempts to load quantized Gemma 4 E2B checkpoints via vLLM on TPU failed due to unimplemented int4 schemes and missing weight tensors, with only the full BF16 checkpoint loading successfully. The author attributed the loading failures to the vLLM loader rather than the checkpoints themselves, noting the QAT exports are architecturally correct, and filed a bug report accordingly.