AI Safety Reviewer Rejected Valid Data Fixes While Passing Its Own Benchmark

A data-catalog governance system called ARGUS uses an AI Arbiter to review proposed metadata repairs before they are applied, acting as a safety checkpoint. During a live sweep, the Arbiter rejected 86 of 112 proposals, including obvious PII classifications like cust_first_name, leaving PII governance at zero percent completion. The root cause was an ambiguous prompt that a weaker AI model interpreted too literally, demanding external proof that a column named cust_first_name actually stored a first name. Compounding the problem, the regression test was skewed toward rejection cases, allowing a reviewer that refused everything to score 75 percent and appear functional. The developer resolved the issue by rewriting the Arbiter's instructions to explicitly distinguish between evidence derivable from schema and lineage versus facts requiring outside corroboration, while also accounting for the costs of both false approvals and false rejections.