SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

High Benchmark Scores Don't Always Mean Better AI Performance, Tests Suggest

0
·4 views

A developer at DEV Community investigated whether Anthropic's Opus 5 model truly outperforms Fable 5 after noticing that improved benchmark scores did not match real-world experience. The author defines 'benchmaxing' as optimizing AI models specifically to score higher on evaluations, which can diverge from actual usefulness when the metric stops reflecting real-world needs. Three distinct issues were identified: test familiarity and potential overfitting, selective reporting of only favorable results, and a gap between evaluation metrics and practical productivity. A METR study of 246 developer tasks found AI tools actually increased completion time by 19%, despite participants believing they had saved time, illustrating why benchmarks alone can mislead. The author's own probes found notable failures in Opus 5 but did not confirm a clear general superiority for Fable, concluding the reality is more nuanced than either praise or accusation.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

How one developer built on-demand PNG card generation inside a Cloudflare Worker

A developer built Commit Archive, a GitHub repo yearbook that generates custom Open Graph and contributor portrait cards as PNGs entirely within a single Cloudflare Worker. The rendering pipeline uses satori to convert layout trees into SVG and resvg-wasm to convert SVG into PNG, with warm-isolate render times averaging 56–82 milliseconds per card. Four notable issues emerged during development: WebAssembly instantiation restrictions on Workers forced a downgrade to satori 0.15.x, a native fetch binding caused illegal invocation errors in queue consumers, GitHub's contributor stats endpoint returned 202 responses for up to 15 minutes triggering retry exhaustion, and a shared free-plan request quota caused Cloudflare error 1027 after a separate Worker on the same account exceeded the 100,000 daily request limit. Each problem led to targeted fixes, including wrapping fetch in an arrow function, publishing jobs without line counts on first retry, and separating Workers across accounts.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

BackToSchool gets standalone accounts, makes NixAmp an optional broadcast add-on

The educational platform BackToSchool (backtoschool.help) has separated its account system from NixAmp, meaning teachers no longer need a NixAmp account to create and run classes. Teacher profile details — name, photo, and bio — are now stored once on the account and automatically applied to every class created, eliminating repetitive form-filling. Users can optionally connect NixAmp via an OAuth 2.1 flow with PKCE, which lets them select live broadcast servers and channels directly from the class form without manually copying links. A new "Teach this on backtoschool.help" button on NixAmp.com allows hosts to instantly convert a live room into a classroom with one click. Both updates are live now on backtoschool.help and nixamp.com.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

LiteLLM Auth Bypass Flaw Exposes AI Gateway Credentials to Unauthenticated Access

A critical authentication bypass vulnerability in LiteLLM's MCP Streamable HTTP endpoint allowed unauthenticated users to access the gateway by sending a forged or invalid Bearer token, which triggered a fallback to an empty authentication object instead of rejecting the request. Because LiteLLM proxies centrally store API keys for multiple AI model providers and broker connections to databases, code repositories, and internal APIs, the flaw gave attackers broad access to whatever resources the gateway was configured to reach. Security researchers from Wiz and Microsoft reported in September 2026 that the vulnerability was chained with two other flaws — CVE-2026-42271 and CVE-2026-48710 — to achieve unauthenticated remote code execution, with the attack chain linked to the Qilin ransomware group. Attackers were also observed recovering the LiteLLM master key directly from process memory and concealing mining binaries within AI-related directories. Operators are advised to upgrade to LiteLLM version 1.84.0 or later, rotate all exposed credentials, and audit the tools and resources their MCP gateway exposes.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Stolen OAuth Refresh Tokens Can Survive Password Resets, Leaving SaaS Accounts Exposed

In SaaS environments, attackers who steal OAuth refresh tokens can maintain persistent account access even after a victim resets their password, because many identity providers do not automatically revoke tokens on password change. Refresh tokens are long-lived credentials — valid for days or months — that allow an attacker to continuously generate new access tokens without ever re-entering a password. Common theft vectors include infostealer malware, misconfigured logging pipelines that capture authorization headers, and leaked CI/CD secrets. Security experts recommend that organizations pre-document revocation capabilities for each identity provider, shorten refresh token lifetimes, and enable rotation with reuse detection before an incident occurs. Stronger protections such as sender-constrained tokens (e.g., DPoP) can render stolen tokens useless by cryptographically binding them to a specific client key.

0 comments Read more at DEV Community