Frozenset Architecture Keeps LLM Critic Reliable Despite 100% Verdict Inconsistency

A software developer building an LLM-based plan critic found that the model returned a different verdict and reasoning on every one of five identical test runs, yielding a label flip rate of 1.0. Despite this complete non-determinism, no defective plan ever passed through undetected, because safety-critical checks are handled by deterministic code gates rather than the LLM itself. The system separates two failure modes — dangerous under-claiming and noisy over-claiming — and assigns each to a different non-LLM authority. A hardcoded frozenset of eligible blocker categories automatically downgrades any LLM verdict that flags issues outside a defined set of genuine safety concerns. This design, refined after earlier prompt-engineering attempts failed, ensures that LLM inconsistency affects only noise levels, not the core safety contract.