AsyncGRPO Cuts GPU Idle Time in RL Training by Decoupling Slow Environment Checks

Most reinforcement learning teams training models on complex real-world tasks face a severe GPU underutilization problem, where fast inference finishes in 1–2 seconds but CPU-side environment checks like compilers and simulators take 15–60+ seconds. Under traditional synchronous GRPO, GPU clusters sit completely idle for 75–85% of total training time while waiting for these verifiers to complete. AsyncGRPO addresses this by breaking the stop-and-wait loop into an asynchronous streaming pipeline that decouples rollout generation, environment execution, and policy updates. This allows GPUs to begin generating the next batch of rollouts while CPUs are still evaluating the previous one, eliminating the idle bubble. The approach also uses bounded staleness via importance sampling to ensure that policy gradient correctness is preserved despite the version lag introduced by asynchronous execution.