Dev builds reproducible harness to benchmark AI coding models on real codebases

A developer has published a lightweight, language-agnostic evaluation framework designed to test AI coding models against a team's own codebase rather than relying on public benchmarks or anecdotal comparisons. The harness organises tasks as directories, each containing a prompt, a relevant code snapshot, and a shell-based verification script, so correctness is judged by the project's own tests and linters. A minimal Python runner applies model-generated patches to each task in an isolated temporary environment and records a simple pass or fail result. The tool deliberately separates patch generation from verification, allowing teams to compare outputs across multiple models or re-run checks at any time. The author argues this approach produces more defensible, context-specific answers about model usefulness than cherry-picked examples or leaderboard scores.