Open-Source Tool CauterRule Finds Over Half of AI Agent Rule Candidates Inconclusive

CauterRule, an open-source tool released on GitHub and PyPI, learns standing rules from repeated AI agent failures by extracting lessons from interaction trajectories and replay-testing them. A field test covering 394 trajectories across four AI models generated 1,538 rule candidates, of which 52.9% were deemed inconclusive — more than the pass and fail results combined. Among the models tested, the cloud-based GPT-4o-mini had the highest inconclusive rate at 62.9%, exceeding both local models, while meta-llama/llama-3.1-8b-instruct had the lowest at 42.9%. The tool's replay engine uses substring and token-overlap heuristics to evaluate rules, and returns an inconclusive verdict when overlap scores fall into a gray zone between pass and fail thresholds. The findings highlight that ignoring inconclusive results — as most benchmark reports do — means discarding the majority of evaluation data and potentially drawing flawed conclusions.