AI Prompt Optimizer Finds Real Gains but Fails Statistical Gate in Repeated Tests

A developer building a self-improving AI prompt agent found that genuine prompt edits consistently failed to achieve statistical significance under a strict permutation-test gate. In version 0.1.0, an edit fixed 4 tasks and broke 1 out of 26, yielding a mean delta of 0.115 but a p-value of 0.23 — well above the 5% threshold required for promotion. Expanding the test corpus to 40 tasks in v0.2.0 was expected to increase statistical power, but the analyzer's edits still moved only 1–2 tasks, shrinking the effect size from 11.5% to 2.5%. The core problem identified was not task count but the analyzer's inability to produce edits that move a sufficient number of tasks without introducing regressions elsewhere. The experiment illustrates that more data only improves statistical power when the underlying effect size remains constant — a condition the AI agent failed to meet.