Developer Builds Repeatable C++ Grader to Rigorously Evaluate AI Coding Assistants

A software developer frustrated with benchmark screenshots built a small automated grader to objectively evaluate AI coding assistants on real C++ tasks. The grader subjects each model-generated fix to three mechanical checks: whether the code compiles under the project's existing strict flags, whether its output matches a hidden expected result, and whether edits stayed within the permitted scope. Tasks are drawn from actual past defects, such as off-by-one errors, dangling references, and missing virtual destructors, keeping the corpus small but meaningful. Running multiple samples per task rather than relying on a single completion revealed that some models oscillated between clean fixes and warning-introducing rewrites, making consistency a more useful signal than peak cleverness. The author notes the approach is language-agnostic in method and that the first common failure point was warnings-as-errors, not algorithmic correctness.