AI Authored Policy Boundary Tests With 98% Validity Across 50 Repeated Runs

A developer running Study 011 on the open-source Judgment Pack Specification project tasked an AI model with independently authoring test cases for a synthetic vendor-screening policy containing sanctions rules, country restrictions, and risk thresholds. The same blinded authoring prompt was run 50 times using a fixed model, prompt, and policy to measure how reliably the AI could identify critical decision boundaries. Of the 50 runs, 49 passed pipeline validation checks, and all 49 valid runs covered all six preregistered boundary classes without exception. Every valid run produced exactly 16 accepted records, with all 784 accepted records agreeing with the reference policy's intended logic, and no two valid completions were identical. The researcher cautioned that while the results were striking, they apply only to this specific model, prompt, and synthetic policy and should not be treated as a universal benchmark for AI-assisted policy testing.