Developer's own eval tool finds multi-agent AI harness costlier and no better

A software developer built a benchmarking tool to validate the assumption that multi-agent AI harnesses — using a planner, multiple drafters, and a judge — produce better results than a single drafter. Testing across 20 coding tasks, the single-drafter setup scored 95% at $0.031 per task, while the four-call panel scored 80% at $0.692, making it 22 times more expensive and over 8 times slower. However, the tool cautioned that the result was statistically inconclusive, as only 3 of 20 tasks were decided and that sample is too small to reach significance. The developer noted that most evaluation tools would have simply ranked the two scores and implied a clear winner, which would have overstated what the data actually supports. The tool was designed with deterministic scoring, paired per-task comparisons, and an exact sign test to avoid misleading conclusions from small benchmark suites.