Study finds over 88% of AI agent instructions are unverifiable in practice

A developer auditing his own AI coding workspace discovered that the vast majority of instructions given to AI agents cannot be independently verified from a repository alone. To test whether this was a broader problem, he built a tool and analyzed eight public agent-instruction collections, covering 1,332 instruction units and 17,611 individual instructions. The analysis found that only a median of 11.1% of procedural instructions are checkable — meaning a reviewer could confirm from the repo whether they were followed — with the range spanning 2.2% to 22.9% across collections. Five of the eight collections required no output artifacts whatsoever, while 43% of artifacts mandated by the remaining three had never actually existed in any commit. The author concludes that most AI agent instructions function not as enforceable rules but as unverifiable claims, with no mechanism to confirm compliance either way.