AI Code Review Benchmarks Are Only as Reliable as Their Bug-Counting Methods

·3 views

A September 2026 benchmark by Entelligence tested two AI code review models, GPT-5.6 Luna and GPT-6 Astra, against 50 public pull requests from well-known open-source projects. Astra detected more bugs with higher precision (96%) but cost nearly 28 times more than Luna, which still achieved 74% precision. The benchmark stood out for publishing its methodology, including a dual-judge verification system where both reviewers had to agree before a finding counted as a real bug. Experts note that most AI tool comparisons omit such details, making precision and recall figures difficult to trust without knowing who labeled the data and how disagreements were resolved. Teams evaluating code review tools are advised to demand transparency on labeling protocols, judge independence, and inter-rater disagreement rates before acting on any benchmark claims.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Finger on iPhone Speaker Was Tricking AI Avatar's Voice Detection System

A voice dialogue avatar that worked correctly on PC was repeatedly cutting itself off on iPhone Safari due to a subtle audio glitch. Covering the phone's speaker with a finger while the avatar spoke disrupted the browser's echo canceller, causing residual sound to leak into the microphone. The voice activity detector (VAD) mistook this leaked audio for human speech and triggered false interrupts every two seconds, though no transcription was produced since the sound contained no words. The fix involved raising the minimum continuous speech duration required to trigger an interrupt — from 0.2 to 0.5 seconds — only during avatar playback, so brief echo artifacts are filtered out without slowing normal turn-taking. Research from LiveKit supported this approach, showing that analyzing roughly 216ms of audio before acting on a VAD signal rejects over half of false interrupt triggers.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Capital One's 2019 Data Breach Was a Misconfigured IAM Role, Not Just an SSRF Flaw

In March 2019, an attacker exploited a misconfigured web application firewall at Capital One to perform a server-side request forgery, retrieving AWS instance metadata credentials via the IMDSv1 service at 169.254.169.254. The critical factor was not the SSRF itself but the overly permissive IAM role attached to the firewall instance, which granted read access to S3 buckets across the entire account, exposing personal data of 106 million people. The breach went undetected until July 17, 2019, when an outside party alerted Capital One through its responsible disclosure channel. Capital One faced an $80 million civil penalty from the OCC in August 2020 and a $190 million class action settlement, with regulators citing failures in risk assessment and internal controls. The incident remains a cautionary study in the principle of least privilege — the encryption in place was rendered ineffective because the same credentials used to access the data could also decrypt it.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Radio Lets AI Agents from Different Providers Collaborate in a Shared Channel

A new tool called Radio functions like a Slack workspace designed specifically for AI agents, giving them a shared channel to communicate and divide work. It enables agents from different providers, such as Claude and ChatGPT, to collaborate on a single project without losing context between handoffs. Currently, many developers manage these handoffs manually by having one agent produce documentation for another to pick up. Radio aims to eliminate that friction by letting agents directly clarify decisions, architecture choices, and task details with each other in real time. The tool could also streamline workflows involving computer-use tasks by allowing specialized agents to coordinate seamlessly within one channel.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How a Missing Data Field Crashed 8.5 Million Windows Machines on July 19, 2024

On 19 July 2024, a faulty configuration update from cybersecurity firm CrowdStrike triggered blue screens and boot loops on approximately 8.5 million Windows devices worldwide, disrupting airlines, hospitals, and other critical services. The root cause was a mismatch between a template defining 21 input fields and integration code that only supplied 20, causing an out-of-bounds memory read in a kernel driver. The defect had gone undetected for four months since February 2024 because all prior deployments used wildcard matches that never required reading the 21st field — a gap that both production and testing environments shared. Independent analysis confirmed the incident was not a cyberattack and that the flaw was not exploitable by malicious actors. CrowdStrike reported roughly 99% of affected sensors restored by 29 July, though recovery required manually deleting a single file on each affected machine, one at a time.

0 comments Read more at DEV Community