Open-Source AI Agent Tests Self-Rewriting Prompts, Rejects 'Mostly Right' Improvements

AgentSelfEdit is an open-source tool that allows an AI agent to rewrite its own system prompt based on execution feedback, then A/B tests edits to promote only statistically proven improvements. The project was tested on a 26-task classification benchmark where the baseline prompt scored just 46%, with the model making systematic errors around urgency detection, keyword over-indexing, and multi-label classification. An LLM-proposed edit added four priority rules, expanding the prompt from 212 to 939 characters, and correctly fixed four of the failing tasks. However, the edit broke one previously correct task, causing the system's quality gate to reject the change despite the net improvement. The project highlights a core challenge in self-improving AI systems: partial correctness is treated as insufficient, since any regression — however small — can disqualify an otherwise beneficial update.