Checking Test Results in Coding Agent Logs Is Harder Than Reading the Exit Code

A developer analyzed about 16,000 public coding agent runs to see whether the final test result could be determined by reading the last test command's exit code. The approach proved unreliable: for one model, 88 runs appeared to end on a failing test, but after corrections only 59 did, since many apparent failures were actually timeouts, signals, or non-test errors. Piped commands can mask the real exit status, and some codes such as pytest's 2 through 5 signal interruptions or the absence of tests rather than failures. The author recommends parsing the test summary line, stripping color codes, and ignoring coverage-gate messages and zero counts, while keeping an 'unknown' verdict for runs where no tests executed. Log readers also need to account for commands like git stash that can make a passing agent run look like a failure.