Neutral AI Code Review Benchmark Scores 16,000 PRs Without Vendor Bias

·16 views

AI research lab Martian has published Code Review Bench, an independent benchmark evaluating AI code review tools using over 16,000 real open-source GitHub pull requests. Unlike vendor-produced rankings, the methodology is fully public and the benchmark code is MIT-licensed, making results reproducible by anyone. The benchmark measures each tool's precision, recall, and F1 score based on whether developers actually acted on a bot's suggestions. Cubic Dev AI topped the overall F1 rankings at 65.7%, followed by GitHub Copilot at 63.9% and Claude at 62.5%, with a relatively narrow spread across the top tools. The leaderboard covers 14 tools and reveals meaningful trade-offs between thoroughness and noise that no single vendor's marketing material reflects.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer builds errval to bring Go-style error handling to TypeScript

A developer who works primarily in TypeScript and Go built errval, a lightweight TypeScript library that brings Go-style explicit error returns to TypeScript projects. The library weighs just 1.86 kB minified and gzipped, has zero dependencies, and works across Node, Bun, and Deno runtimes. It uses a tuple pattern — const [err, value] — where the error is placed first to prevent developers from accidentally ignoring it. Benchmarks show errval handles failure-path requests at 226 ns per request, slower than neverthrow's 198 ns but significantly faster than plain try/catch at 2,623 ns on the failure path. The project is at version 0.1, currently maintained by a single developer, and is available via npm and GitHub.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Git's rerere Feature Can Automatically Reuse Your Past Conflict Resolutions

Git includes a built-in feature called rerere (reuse recorded resolution) that remembers how a developer resolved a merge conflict and replays that resolution if the identical conflict appears again. Once enabled via a single config setting, Git stores a preimage of each conflict and the corresponding postimage resolution under the .git/rr-cache/ directory. By default, rerere writes the saved resolution into the file but leaves it unstaged, allowing the developer to inspect the result before proceeding. An optional autoupdate setting can also stage the file automatically, though the rebase or merge operation still pauses for a manual review. Recorded resolutions can be discarded with git rerere forget, and rerere only triggers when a conflict matches byte-for-byte, so any change to either side of a conflict prompts a fresh resolution.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Undo a Git Commit: The Right Command Depends on Two Questions

Undoing a Git commit has multiple correct approaches, but choosing the right one requires answering two questions first: where does the change currently live, and has anyone else pulled it? For local, uncommitted changes, commands like 'git restore' or 'git restore --staged' are appropriate depending on whether the edit is in the working tree or the index. Committed but unpushed changes can be safely undone with 'git reset --soft HEAD~1', which removes the commit while preserving the staged content. Once a commit has been pushed and shared with others, rewriting history is risky, and 'git revert' is the safer choice as it adds an inverse commit rather than deleting history. Using 'git reset --hard' is the most destructive option, permanently discarding uncommitted changes with no recovery path through the reflog.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Git Bisect Can Pinpoint a Buggy Commit in Just Four Test Runs

Git's built-in bisect tool uses binary search to identify which commit introduced a regression, requiring only a known good commit, a known bad commit, and a test that returns an exit code. In a 12-commit example repository, running 'git bisect run' with an automated test script narrowed down the culprit in just four steps without any manual intervention. The offending commit, labelled 'Perf: simplify addition loop', had stripped minus signs from numeric arguments, silently converting all negative numbers to positives. Git bisect works by checking out the midpoint of the commit range, running the test, and halving the search space based on the result — a process that scales logarithmically regardless of how many commits are involved. For reliable results, the test script must be independent of files added after the known-good commit, and endpoints must be verified, as an incorrect 'good' baseline will lead bisect to the wrong answer.

0 comments Read more at DEV Community