How to Build a Reproducible Harness to Compare AI Coding Models on Your Own Repo

·1 views

Generic AI coding model benchmarks often fail to reflect real-world codebases, which have unique build systems, legacy code, and test suites. A developer has shared a roughly 120-line shell and Python harness that lets teams evaluate models against their own repositories using actual test results. The approach mines a project's git commit history for small, self-contained bug fixes or feature additions, then asks each model to reproduce those changes without seeing the original solution. Models are scored across three axes: correctness, edit locality, and iteration cost. The author notes that repetition across runs matters more than most expect, as variance within a single model can exceed the gap between different models entirely.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How to Customize Hugo PaperMod Theme Without Forking It

A developer has documented a method for deeply customizing the Hugo PaperMod theme without forking it, keeping the theme as an updatable submodule. The approach relies on Hugo's file resolution order, which prioritizes site-level files over theme files, enabling full overrides through a small set of custom layouts, data, and CSS files. A custom homepage is created via a single layouts/index.html file, with all copy stored in language-specific YAML data files to support multilingual sites cleanly. Visual changes such as color palette and typography are applied through CSS variables in the assets/css/extended/ directory, requiring no edits to theme files. The guide also flags common pitfalls, including broken dark mode caused by outdated body.dark selectors that no longer work with PaperMod's current data-theme attribute approach.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Karpathy's LotR Demo Shows AI Agents Can Handle Hours-Long Tasks Autonomously

AI researcher Andrej Karpathy recently demonstrated Claude Opus's capabilities by feeding it a million-token context, the opening paragraph of The Lord of the Rings, and a prompt to generate a procedural 3D scene in Three.js. The model worked autonomously for roughly two hours, producing 5,500 lines of code covering polygon placement, camera paths, and animation — at a total cost of about $10. The demo signals a shift in how work can be delegated to AI: rather than breaking tasks into small, supervised chunks, developers can now hand over full material and intent and await a complete result. The key human skills in this new paradigm become crafting the input brief — deciding what context goes in — and defining clear acceptance criteria before the run begins. Experts caution, however, that autonomous long-session output remains unsuitable for production software where errors affect real users and require ongoing maintenance.

0 comments Read more at DEV Community

ProgrammingHacker News ·

The software group Palantir paid just £2M in corporation tax in the UK in 2024

Article URL: https://www.theguardian.com/technology/2026/aug/05/palantir-corporation-tax-uk-2024-accounting-practices Comments URL: https://news.ycombinator.com/item?id=49180535 Points: 6 # Comments: 2

0 comments Read more at Hacker News

ProgrammingDEV Community ·

10 CLI Tools Developers Should Add to Their Workflow in 2026

A developer writing for DEV Community has highlighted 10 command-line interface tools that improve productivity and streamline daily development tasks. The list includes widely used tools such as Git, Docker CLI, GitHub CLI, kubectl, and HTTPie, alongside utilities like jq, ripgrep, fd, and just. Apidog CLI, used for running API test scenarios and managing environments within CI/CD pipelines, was singled out as the most impactful addition to the author's workflow. The author argues that moving repetitive tasks — including API testing, deployments, and version control — into the terminal makes them easier to script and automate. The common thread across all recommended tools is their ability to integrate with automation pipelines and maintain consistency between local development and CI/CD environments.

0 comments Read more at DEV Community