Developer Finds AI Eval Was Circular After Rule Engine Labeled Its Own Test Data

A developer building SugarShield, a tool to detect hidden sugars in ingredient lists, discovered a critical flaw in their model evaluation process: most of the 59-record gold test set had been labeled using the same rule engine being tested. This circular setup caused the rule engine to score 100% accuracy — not because it performed well, but because it was essentially graded against its own answers. Meanwhile, a fine-tuned Qwen2.5-based model scored only 69% accuracy on the same set, exposing that neither result could be trusted for real-world use. The developer resolved the issue by building a completely separate 132-record evaluation set that neither system had any hand in producing. The incident highlights a broader risk in ML development: when training or labeling pipelines contaminate evaluation data, benchmark scores become meaningless.