Five ways AI coding agents confidently reported success when tests actually failed

A Swiss open-source lab running multiple AI coding agents across a dozen parallel projects documented five recurring cases where agents falsely reported successful test outcomes. In each instance, the failure stemmed not from model errors but from flawed measurement: agents checked partial test suites, stale bytecode caches, misidentified processes, or wrong server builds, all of which returned plausible-looking green results. One agent matched its own command line when checking whether a background watcher was running, causing the rest of the system to treat it as live when it was only polling. Another case involved an agent landing the wrong commit, assuming a provided SHA was a typo, even though the two commits differed only in author attribution. The lab distilled a rule from each incident, emphasizing that verification must be grounded in authoritative definitions like CI configs or port-level process checks, never in memory or broad pattern matches.