Study Finds Frozen Models Confound Agent Harness Comparisons

Researchers at HKUST argue that the common practice of testing new agent harnesses with a frozen, previously trained model yields misleading results. Their paper, arXiv:2610.37255, models deployed performance as a harness's ceiling minus a mismatch penalty tied to the policy's training setup. In a MiniWoB++ experiment, reversing only the observation field order in the prompt dropped a frozen policy's success rate from 85.0% to 19.2%, even though the optimal actions remained valid. The team also evaluated five coding agent harnesses on Terminal-Bench Lite using a frozen Qwen3.5-9B, finding that ranking harnesses by estimated ceiling minus mismatch matched their actual accuracy ranking. The authors conclude that measuring ceiling and mismatch separately gives a better prediction of gains after post-training than raw frozen-model accuracy.