Developer Builds Git-Based Harness to Benchmark Coding AI Models on Real Work

A software developer has published a reproducible testing harness that evaluates free-tier coding AI models against actual tasks drawn from a project's own git history, rather than relying on public benchmarks like HumanEval. The system mines bugfix commits from git logs, reconstructs the pre-fix codebase, and uses the original commit message as a prompt for the model to solve. Each model-generated patch is then graded on three criteria: whether it applies cleanly, whether it passes the associated tests, and whether it modifies files outside the intended scope. The author argues that public benchmarks are both saturated and unrepresentative of real-world coding tasks, making them unreliable for day-to-day tool selection. The harness produces fifteen data points across five tasks and outputs results to a CSV, offering a low-cost, project-specific alternative to generic model evaluations.