Study Finds Small Language Models Fail Eligibility Thresholds for Agent Microtasks

A paper on arXiv (2610.00025) examines whether small language models can handle narrow, high-frequency microtasks in agent harnesses, such as auto-approving shell commands, selecting tools, and managing memory. Testing Qwen3 models from 0.6B to 8B and Llama-3.x found that none of the 16 configurations met the confidence-anchored threshold (TAU = 0.95) set by a non-LLM baseline. Quantization to 4-bit precision did not change the outcome, and the gap appeared to scale with model size rather than numerical precision. The authors recommend placing a baseline that meets the threshold in front of small models and invoking them only when that baseline fails. An accompanying open-source, AGPL-3.0-licensed harness implements this fail-closed approach and reports 0 of 16 malicious inputs passing the benchmark.