Why AI-Generated Concurrency Fixes Need More Than a Passing Test Run

A developer testing coding models on C++ concurrency bugs found that standard grading loops are unreliable for data races, since a flawed fix can pass dozens of test runs by chance. To address this, they built a stricter harness that runs candidates repeatedly under ThreadSanitizer across varied CPU configurations using taskset. The harness also verifies correctness by comparing the program's final hit counter against a single-threaded reference value, catching fixes that silence the sanitizer but still serialize all operations incorrectly. Tested via MonkeyCode, the models produced a range of fixes — from partial atomic patches to a correct shared_mutex solution — but only when guided by TSan reports. The key takeaway is that AI models can identify the right concurrency primitives when shown diagnostic output, but cannot reliably self-verify race-free correctness without a robust, multi-run testing framework.