MMLU-Pro still grades 52 questions against MMLU answer keys experts flagged as wrong

·2 views

A research effort led by AI agents at Munim, Inc. cross-referenced expert corrections from MMLU-Redux 2.0 with the MMLU-Pro benchmark. Of 5,700 MMLU test questions the experts reviewed, 106 were marked as having incorrect answer keys. Matching these to MMLU-Pro found 55 flagged questions still in the benchmark, and 52 of them still use MMLU's original, disputed key. The affected questions are spread across subjects, with a notable cluster in college chemistry. Because these make up only about 0.43% of MMLU-Pro, overall model scores shift only slightly, though the effect is larger in certain subject slices.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

AI Can Generate Code, but Engineers Are Still Needed to Make Software Work

Tech leaders Marc Benioff of Salesforce and Mark Zuckerberg of Meta have predicted that AI will transform software engineering, with Benioff citing productivity gains for possibly limiting 2025 hiring. The article argues that generating code is only one part of building dependable software, since engineers must grasp user needs, integrate changes into complex systems, and verify outcomes. A 2025 METR study of 16 experienced developers found they took about 19% longer on tasks with AI tools, even though they believed the tools had sped them up. Google's 2024 DORA research linked higher AI adoption to lower delivery stability across the organizations it studied, though it did not establish causation. The piece concludes that AI will reshape engineering work but will not remove the need for human judgment, review, and responsibility.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why a clean static analysis scan can still miss real vulnerabilities

A security series author ran a free static analyzer on their own service's code and got zero findings with its default rule set. A second run using a JavaScript-specific rule set flagged the post-login redirect logic, and the underlying flaw went unnoticed by every Semgrep rule set they tried. An AI code reviewer caught the issue when the fix was tested, and it has since been patched. The author's takeaway is that zero findings does not guarantee safety, so developers should layer different checking methods, verify every hit, and rerun scans regularly. The piece also notes that static analysis can reveal secrets such as API keys left in git history, including a deactivated AWS key found with Gitleaks.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Remote Teams Can Choose the Right Online Whiteboard Tool

Online whiteboards have replaced physical walls for remote and hybrid teams, supporting brainstorming, retrospectives, diagrams and design work. The article argues that tools look alike in demos, so differences only emerge after weeks of real use with large boards and many participants. It recommends first listing the main situations where a team will use a whiteboard, then ranking them to decide which trade-offs are acceptable. Teams should test the canvas with realistic content, including performance on large boards, text handling, connectors, frames and recovery options. Collaboration features such as cursors, facilitation tools, guest access and asynchronous commenting also deserve trial sessions before a decision is made.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Researcher Claims Flaw in Vaani Voice AI Let Him Hijack Agents Across Tenants

A security researcher describes finding a flaw in Vaani, a voice AI agent platform that won Product Hunt's top daily spot, at a company hackathon. He says a single HTTP request from a free account could take over AI agents belonging to other customers, redirecting their outbound calls to read attacker-supplied prompts. The issue reportedly stemmed from predictable agent identifiers that were not properly protected by access controls. The researcher says he demonstrated the hijack without accessing private data and that the vendor has since patched the problem. The account highlights risks in multi-tenant voice AI systems that place outbound phone lines under automated agent control.

0 comments Read more at DEV Community