Developer finds shared blind spot in two AI safety architectures built to catch LLM errors

A developer built two AI oversight systems this year — PlannerCritic and AdversarialDebate — designed from opposite angles to detect when large language model judgments fail. PlannerCritic uses a deterministic code-based gate layer paired with an LLM critic to audit and revise plans until they meet a safety contract, while AdversarialDebate pits two LLMs against each other in a structured, independent review protocol. Both systems performed well on metrics, but the developer later identified a critical shared flaw: failures in each system caused dashboards to show improvement rather than raising alerts. In PlannerCritic, a gate that silently stops firing reduces blocker counts, making plans appear safer when they may not be; a community reader drew the parallel to a Kubernetes incident where broken deployments went undetected because a legacy pod kept health checks green. The developer concluded that robust architecture alone is insufficient — systems designed to catch model errors must also be able to detect when their own safety mechanisms quietly stop working.