CauterRule v0.3.0 Exposes Flaw in AI Rule Validation: Text Matching Masks True Accuracy

CauterRule, an open-source tool that converts repeated AI agent failures into reusable standing rules, released version 0.3.0 with findings from a field test covering 40 corpora and 4,768 trajectory runs across two cloud models. Developers discovered that the tool's replay gate, which decides whether an extracted rule gets promoted, relies on lexical text similarity rather than verifying whether a rule would actually change an agent's outcome. In one documented case, a correctly extracted git rule was demoted to 'inconclusive' because unrelated successful trajectories happened to share the token 'git' with the rule's trigger phrase. The team also found that extraction accuracy, measured by token-F1 against ground-truth rules, scored only around 0.50–0.58, not due to wrong outputs but because the model rephrases correct answers that a token comparator penalises. The release reframes the problem as two distinct challenges — rule extraction quality and replay verification validity — pointing to separate fixes needed for each.