Developer Finds Model Size Was Never the Issue in Self-Improving AI Agent Experiment

A developer building a self-improving AI agent tested four language models — ranging from 1B to 30B parameters — across 4,150 LLM calls, hoping a larger model would generate better prompt edits. All four models failed to produce a single promotable edit, with each converging on the same narrow region of the prompt rather than exploring new approaches. The most capable model tested, Mistral 24B, showed directional improvement in three of five iterations but never reached statistical significance. The developer concluded the core problem was not model capability but the search strategy, which lacked any mechanism to step back and explore fundamentally different edit types. Failures were concentrated in multi-label classification, ambiguous categorisation, and keyword over-indexing, issues that minor prompt rewording consistently failed to address.