OpenAI Audit Finds 30% of SWE-Bench Pro Coding Tasks May Be Flawed

·1 views

OpenAI published an audit of SWE-Bench Pro on July 8, 2026, estimating that approximately 30% of its benchmark tasks may be broken or invalid. The finding undermines a core assumption behind AI coding leaderboards — that every task in the scoring denominator is a fair and equally interpretable test. Experts argue the appropriate response is not to abandon benchmarks but to version task validity, flag disputed cases, and report how scores shift across different denominators. A proposed framework recommends publishing separate scores for verified-valid tasks versus all attempted tasks, along with a sensitivity range showing how unresolved cases could affect results. The approach also calls for strict data provenance and consistent task sets when comparing models, to prevent misleading performance claims.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Indie Hacker Spent Two Months Avoiding One Question: Will You Pay for This?

An indie hacker shared a candid account of how engineering habits led him to design a flawed product validation experiment. In May, he launched an AI tool with free credits instead of charging users, assuming high trial numbers would signal real demand. After two months of building, marketing, and waiting, he had only around 20 sign-ups — none of which revealed whether anyone would actually pay. Conversations with experienced founders helped him realize he had measured willingness to try for free, not willingness to pay — two fundamentally different things. His key takeaway: avoiding the uncomfortable act of asking someone for money can quietly cost founders months of wasted effort.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Treating Remote Contractors Like Staff Employees Can Trigger Serious Legal Liability

Companies that manage remote developers like full-time employees risk misclassification under U.S. tax and labor law, regardless of what their contracts say. Three separate legal tests — the IRS common-law test, state ABC tests, and the DOL's economic-reality test — determine worker status, and they can reach different conclusions on the same facts. Behavioral indicators like fixed hours, company-issued tools, and integration into core workflows signal employee status to regulators, overriding any signed contractor agreement. Misclassification can result in back taxes, unpaid employer FICA contributions, and wage claims, with penalties applying even when the error was unintentional. Businesses seeking to reduce classification risk are increasingly routing contractor relationships through a Contractor-of-Record service rather than managing compliance internally.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

BrowserAct Offers a Structured Browser Layer for AI Agents Beyond Playwright

As AI systems increasingly perform complex web-based tasks, traditional automation tools like Playwright and Selenium are proving insufficient for autonomous agents that must navigate unfamiliar websites independently. These script-based tools offer low-level browser control but lack the contextual understanding AI agents require, forcing models to handle all page-interpretation logic themselves. BrowserAct is a command-line browser automation tool designed specifically to give AI agents a structured, readable web environment rather than raw browser APIs. The tool supports parallel account isolation and persistent session management without manual configuration, and its partners include Google Cloud and Oracle. Installed via a CLI, BrowserAct serves versioned skill content to AI coding environments like Cursor, ensuring instructions remain consistent with the installed version.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How SBA-7a Lenders Are Detecting Document Fraud After PPP Losses

The U.S. Paycheck Protection Program exposed what analysts now call the largest controlled experiment in small-business document fraud, with the SBA Inspector General estimating losses in the tens of billions of dollars. Unlike synthetic identity fraud, the dominant attack involved real borrowers using consumer tools like Adobe Acrobat and online PDF editors to alter bank statements, tax returns, and payroll records. Post-PPP, fintech and SBA-7a preferred lenders — including Funding Circle, Bluevine, OnDeck, and Live Oak — have converged on shared detection playbooks targeting recurring fraud patterns in stipulation documents. Key forensic signals include mismatched creation and modification timestamps, multiple document revision layers, and producer-field fingerprints left by consumer PDF editors. However, experts caution that file-level forensics have real limits, particularly with smaller community banks whose export tools can mimic signs of tampering.

0 comments Read more at DEV Community