OpenAI retracts SWE-Bench Pro endorsement after audit finds 30% of tasks flawed

·2 views

OpenAI has withdrawn its recommendation of SWE-Bench Pro, a coding benchmark it had promoted as a replacement for the previously discredited SWE-Bench Verified. An internal audit by OpenAI's research team found that approximately 200–249 of the benchmark's 731 tasks are broken, representing roughly 30% of the dataset. Reviewers identified four recurring failure types: overly strict tests, underspecified prompts, low-coverage tests, and misleading prompts that steered models toward incorrect behavior. The root cause is structural — the benchmark was built from real GitHub pull requests, which are designed for human collaboration rather than clean model evaluation. OpenAI is now calling for new benchmarks built by experienced software developers specifically to assess model capabilities, though no replacement has been announced yet.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

NBomber Studio 0.8.2 Released With Stability Fixes and Air-Gap Support

NBomber Studio version 0.8.2 has been released as a patch update focused on improving stability. The new release enables the tool to run fully air-gapped, meaning it no longer needs to reach out to public content delivery networks. A bug causing the Active Sessions table to display outdated rows has also been resolved. Additionally, a fix was applied to the C# script validator, which previously would hang when processing large build output.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Anthropic Proposes CVE-Style Severity Scale to Standardize AI Jailbreak Risk

Anthropic has introduced the Cyber Jailbreak Severity (CJS) scale, a five-tier framework ranging from CJS-0 (informational, no real threat) to CJS-4 (critical, expert-level attack enablement), designed to bring consistency to how AI jailbreak risks are assessed and communicated. The framework was published alongside the release of Claude Fable 5 and is positioned as an industry standard that AI developers and governments can use to discuss jailbreak severity in common terms. Jailbreaks are graded across four dimensions: capability gain, breadth of applicability, ease of weaponization, and discoverability, with severity bands that scale exponentially. Anthropic has also outlined a four-category classifier for cybersecurity use cases, from unconditionally blocked requests like ransomware development to permitted activities like secure coding and incident response. The company has launched a HackerOne bug bounty program for Fable 5 jailbreak submissions and is openly soliciting feedback on the CJS framework, signaling an intent to build a shared industry standard rather than an internal policy.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

OpenAI GPT-5.6 Sol rivals Claude Fable 5 intelligence at one-third the cost

OpenAI released its GPT-5.6 model family this week, comprising three tiers — Sol, Terra, and Luna — targeting different price-performance trade-offs. According to benchmarks by Artificial Analysis, the flagship GPT-5.6 Sol scores 59 on the Intelligence Index, just one point below Claude Fable 5, while costing roughly one-third as much per task. In coding agent evaluations, Sol leads all models tested on the new Artificial Analysis Coding Agent Index, scoring 80 and outperforming Claude Fable 5 at approximately 40% lower cost. The release also introduces cache-write pricing for OpenAI, charging 1.25 times the base input rate for cached tokens — a structure already used by Anthropic that reflects real infrastructure costs. The entry-level Luna tier, priced at $1 per million input tokens, is positioned as a cost-efficient option that outperforms several comparable models including GPT-5.5 and Gemini 3.5 Flash.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Over half of text-to-SQL benchmark answers are wrong, study finds

A UIUC research team audited leading text-to-SQL benchmarks and found that 52.8% of BIRD Mini-Dev annotations and 62.8% of Spider 2.0-Snow annotations contained incorrect answer keys, as confirmed by human SQL experts. The errors stemmed from annotators misreading unfamiliar database schemas, applying wrong domain knowledge, and misinterpreting ambiguous questions. When just 100 examples were corrected and 16 open-source agents re-evaluated, relative performance scores shifted by up to 31% and leaderboard rankings moved by as many as 9 places. In response, one developer proposed inverting the benchmark-building process by declaring ground-truth answers as constraints first and then generating a database guaranteed to satisfy them. This approach eliminates annotation error by ensuring expected query results are mathematically built into the data rather than derived by humans reading existing rows.

0 comments Read more at DEV Community