Open-Source AI Self-Editing System Rejected Every Prompt Improvement It Proposed

A developer built AgentSelfEdit, an open-source tool that autonomously rewrites its own AI system prompts based on execution feedback and A/B testing. The system uses a six-check deterministic statistical gate — with no LLM involvement — to decide whether a proposed prompt edit is good enough to promote. Across 15 iterations and over 4,000 LLM calls, the gate rejected every single proposed edit. During testing, an apparent success was traced to a two-line bug where the confidence threshold was set to p < 0.95 instead of the correct p < 0.05, meaning near-random results were passing as improvements. Once fixed, the gate correctly rejected all edits, which the developer argues demonstrates the system working as intended rather than as a failure.