AI Capture-the-Flag Tournament Reveals Model Size Is Not the Key to Hacking Skill

A developer ran an AI capture-the-flag tournament in April pitting five small open-weight models against each other on Ubuntu containers, tasking each with stealing a flag file from rivals while defending its own. Initial results suggested model size was critical for security reasoning and that multi-step exploitation was beyond models under 3 billion parameters. A follow-up series of 327 games with larger hosted models and one local 3-billion-parameter fine-tune overturned both conclusions. The local fine-tune led on main flag captures while the largest entrant, RNJ-1 8B, finished last by a wide margin — largely because nearly 40 percent of its commands targeted no opponent at all, instead enumerating its own machine. Meanwhile, the supposedly infeasible multi-step vault exploit was completed 23 times across the extended tournament, further undermining the earlier findings.