Developer Rebuilds LLM Verification Pipeline After Commenters Spot Key Architecture Flaw

A developer running a multi-part experiment series on LLM output verification discovered a fundamental design flaw after readers pointed out that simple deterministic code checks were being skipped entirely. Commenters Alexey and Manuel noted that obvious garbage outputs — such as placeholders, punctuation-only strings, and zero-test passes — could be filtered cheaply with regex and keyword rules before any LLM call was made. The author confirmed that all four garbage scenarios in their test set were catchable by code alone, meaning prior experiments had wasted LLM budget on inputs that never required semantic judgment. This led to a redesigned three-layer pipeline: deterministic code filters handle shape and contract checks first, with the LLM reserved only for genuinely ambiguous residual cases. A new prototype, Experiment F, tested the revised architecture across 38 scenarios to verify whether the restructured pipeline performed as intended.