AI Agent Flags Code Issue but Misclassifies It, Exposing Limits of Automated Analysis

A developer building an AI-powered code analysis agent for a hackathon deliberately prioritized honesty controls over core functionality to test how the system handles uncertainty. The agent examined two unfamiliar files and returned a structured finding that cited exact bytes, proposed a bounded fix, and labeled its discovery as CONFIRMS_KNOWN rather than claiming novelty. However, the classification was incorrect: the agent matched the finding to a known condition describing empty arrays returning false success, when the actual code does the opposite by exiting with a hard failure on empty input. The agent also argued a caller could not distinguish exit codes, which was demonstrably wrong given the code's explicit exit-2 versus exit-0 logic. The developer published a full run receipt with hashes, framing the failed run as a concrete demonstration of where automated semantic reasoning reaches its limits.