Four Experiments Show 'Deterministic AI Agent' Claims Fail at the Semantic Layer

A software developer on DEV Community ran four controlled experiments to test the core mechanisms promoted in popular 'production-grade AI agent' articles, which claim deterministic constraints can reliably govern LLM-based agent loops. The three mechanisms tested — lexical-overlap thresholds, temperature-0 evaluators, and phase gates — each proved only formally deterministic, breaking down when applied to real semantic judgments. Lexical overlap alone produced a 50% hard misclassification rate on 30 labeled pairs, including cases where a delete instruction was treated as a continuation of a writing task. The developer also attempted an upgraded fix to address these failures, but that too did not hold up under measurement. While the broader direction of wrapping LLM uncertainty in structured constraints is acknowledged as sound, the article warns that treating unvalidated mechanisms as solved engineering is misleading and potentially incident-grade in production.