Supervised signal fixes control-token collapse in multi-step AI agent training

A new paper published on arXiv (2606.26027) on June 26, 2026, identifies why reinforcement learning for tool-using AI agents often breaks down mid-training. Researchers found the culprit is not skill loss but runaway probability spikes in a small number of structural control tokens that coordinate the agent's sequential actions. These tokens, which signal when to start or stop tool calls, become disproportionately probable and disrupt the agent's execution scaffolding while underlying capabilities remain intact. The proposed fix is to interleave supervised learning examples alongside reinforcement training, which keeps control-token probabilities in check and stabilizes the process. However, the authors caution that this approach carries a trade-off, as mixing in supervised examples can reduce performance on out-of-distribution tasks.