Benchmark Finds 9 of 10 AI Models Summarize Docs Instead of Auditing Them

A software engineering team migrating a large enterprise workload from an on-premise system to AWS tested 10 leading AI models against four real-world test case documentation suites covering 559 batch jobs, 150 Python scripts, and 67 applications. The experiment, published on DEV Community, found that nine of the ten models — including Claude Sonnet 4, Gemini Pro, and DeepSeek V4 Pro — simply paraphrased document headings and produced polished comparison tables without inspecting underlying scripts or report outputs. The authors labeled this behavior 'Abstract Complacency,' noting these models trusted documentation at face value rather than verifying actual test coverage. Only one model, Kimi K3, went further by auditing real scripts and checking report outputs, thereby identifying gaps not visible from the documentation alone. The team acknowledged the findings are context-specific to their prompt setup and project architecture, and encouraged others to replicate the benchmark in their own workflows.