Engineer Achieves 28 TPS on Free Kaggle GPUs Using Tensor Parallelism and CUDA Graphs

A developer built ShardFlow v2.1, a distributed inference system that runs a Qwen2.5-7B language model split across two free Kaggle T4 GPU notebooks located in different US cloud regions. Because a single T4's 16GB VRAM barely fits the model with no room for a KV cache, tensor parallelism was used to divide layers between two nodes communicating via a low-cost AWS EC2 relay. Speculative decoding, where a smaller 0.5B draft model proposes batches of tokens for the larger model to verify, lifted throughput from roughly 5 TPS to 14 TPS by reducing costly WAN round trips. The biggest gain came from applying CUDA Graphs to the draft model's forward pass, collapsing thousands of individual Python-issued kernel launches into a single driver call and cutting draft generation time from 112ms to 25ms. The combined optimizations pushed peak throughput to 28.10 tokens per second across two cloud regions over public internet with near-zero infrastructure cost.