SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

SemiAnalysis launches GDPx, an agentic benchmark spanning 11 professional domains

0
·3 views

SemiAnalysis has released GDPx v1.0, a benchmark that tests frontier AI models on 206 tasks run in live environments across 11 professional fields, from chip design to quantitative trading. The task data comes from 14 partner organizations, and each task has a default budget of 8 hours. Claude Opus 5.5 currently leads the 13-model leaderboard with a 12.0% score, but it is also costly, averaging about $51.41 per task. Under strict grading, no model solved any finance or law task, and most quant trading attempts underperformed a reference strategy. SemiAnalysis plans to release new versions every 3 to 6 months to keep the benchmark current.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

DevOps engineer Zhibek shares AWS interview preparation: Terraform, EKS and CI/CD experience

A DevOps engineer named Zhibek, with over five years of AWS cloud experience, has published a set of interview questions and answers drawn from a real interview for a role at Herc Rentals. Zhibek describes past work at Yelp, where blue-green deployments and automatic rollback reportedly made releases 35 percent faster, and at Zulily, where Terraform modules were built for a private Amazon DocumentDB database. The material also covers daily responsibilities such as monitoring CloudWatch, reviewing pull requests and writing runbooks, along with a list of technical and managerial questions. The post is aimed at candidates preparing for similar DevOps and team-lead interviews.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Why Per-Attempt Timeouts Break Retry Loops and How to Fix Them

A retry loop with a 10-second timeout and three retries can run far longer than a caller expects, because per-attempt timeouts do not limit the total operation time. Backoff delays and pre-request work such as DNS lookups or token refreshes add further latency on top of the attempts themselves. The article recommends starting from an overall time budget and dividing it across attempts, using a monotonic clock to track remaining time. It also suggests propagating a shared deadline through the call stack, as gRPC does, so downstream calls draw from one budget. Tradeoffs to weigh include starved final calls, budget sharing in parallel requests, and server-side work that continues after a client-side timeout, which makes idempotent retried writes important.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Europe's Supermarket Freezers Hit by Remote Temperature Tampering, Cause Unconfirmed

Staff at an ICA Kvantum store in Sjöbo, Sweden, found refrigeration units remotely set to around 50°C overnight on 2 November, forcing them to discard spoiled stock. Coop, Ahold Delhaize, Colruyt and Carrefour Belgium also reported similar unauthorized temperature changes across multiple stores. The incidents follow a reported earlier wave in France three weeks prior, which affected hundreds of supermarkets with substantial food losses. Companies describe the events variously as tampering or an IT incident, and no group has claimed responsibility or been publicly identified. Investigators have not yet confirmed whether the incidents are linked or how attackers gained access to the systems.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Researcher finds AI-agent guardrail flaw that let unapproved actions run, prompting fix

A security researcher found that airlock, a prompt-injection guardrail in the Hermes plugin catalog, could be bypassed during unattended hermes -z turns. The guardrail checked a flag that did not reflect that the session was unattended, so actions requiring approval ran without any. The maintainer fixed the issue in airlock 0.2.3, which now queries the approval layer and covers CLI subagents, and the fix was merged via PR #144871. The flaw was disclosed through coordinated disclosure and published as advisory GHSA-mgf8-gh7g-q4rg. The researcher argues that this kind of flaw reflects a broader design weakness in guardrails that decide at the door based on proxy signals.

0 comments Read more at DEV Community