How a Layered LLM Judge System Catches Misleading AI-Generated Diamond Descriptions

A development team building an AI assistant for diamond retail discovered that standard content filters were insufficient to catch subtly misleading product descriptions, such as a technically accurate but practically false claim about stone clarity. To address this, they implemented three layered quality gates: a preventive code-level pipeline, a deterministic regex-based validator, and a structured LLM judge that returns a detailed triage rather than a simple numeric score. Unlike typical LLM evaluators, their judge categorizes claims by support level and source, keeping tone and factual accuracy as separate measurement axes. The judge is used strictly as a development instrument rather than a runtime guardrail, and its verdicts are schema-validated with automatic retries on malformed outputs. The team emphasizes that the judge itself is fallible and must be continuously calibrated, logging both writer errors and judge errors to iteratively improve the overall system.