Developer builds CI system for AI prompts, uncovers flaws in the tests themselves

A developer built Sentinel, a prompt regression tool designed to automatically detect quality drops when AI prompt files are modified in pull requests. The system uses a multi-agent setup to run evaluation cases against both old and new prompt versions, flagging statistically significant score drops before a merge is approved. During development, several critical issues emerged not in the prompts being tested, but in the evaluation suite itself, including miscalibrated rubrics, a noise floor that made failure impossible to detect, and a self-approval flaw where the agent refused to merge without ever alerting a human reviewer. A particularly revealing test showed that deleting a key fallback instruction caused an empty submission's score to jump from a failing 0.80 to an erroneous 16, demonstrating how broken evals can mask serious regressions. The project highlights that in AI-driven CI pipelines, the eval cases themselves are a critical and defensible asset, requiring the same rigor as the production code they guard.