PoPE Study Finds Error Content May Not Drive Self-Repair in Small Code LLMs

Researchers have developed PoPE (Popperian Placebo-controlled Evaluation), a new methodology to rigorously test whether error feedback genuinely improves self-repair in small code language models ranging from 0.5 to 1.5 billion parameters. The framework uses channel-specific placebos that preserve the structure of error feedback while stripping or scrambling its content, enabling a direct test of whether actual error information drives repair or merely its form does. In prompt-channel tests, a content-ablated placebo unlocked 12 problem units versus 10 for live error feedback, while the weight channel showed a scrambled placebo outperforming both the error-content adapter and a no-intervention baseline. These results were classified as mechanism-null, meaning error content showed no consistently superior effect over placebos in frozen small models. The findings urge developers to rethink the assumption that feeding raw error messages back to small LLMs meaningfully improves repair, and call for stricter placebo-controlled standards in self-repair research.