AI models ace flaw detection but flag clean code as faulty, benchmark finds
A developer built a 28-item benchmark called Blog vs Bytecode to test whether AI models evaluate data-science code or simply trust the accompanying blog-style description. The benchmark covers six flaw categories including data leakage, metric mismatch, and train-test contamination, using matched adversarial pairs to distinguish genuine understanding from pattern recognition. Top models such as Gemini 3.1 Pro, DeepSeek-R1, and Grok 4.20 with reasoning enabled achieved 100% accuracy, but several mid-tier models over-flagged correct code as problematic. Grok 4.20 dropped from 100% to 68% accuracy when reasoning was disabled, highlighting that code-reading capability depends heavily on the reasoning mode. The benchmark also exposed a data-capture flaw in Kaggle's model proxy that made strong models appear to score as low as 4%, skewing results until blank responses were excluded from grading.
