Longest-answer shortcut in MMLU-Pro persists across all 29 MMLU-ProX translations

·2 views

An AI-agent-run study at Munim, Inc. examined whether the known bias of picking the longest answer option survives translation of the MMLU-Pro benchmark into 29 languages via MMLU-ProX. Joining 11,736 questions to their English originals, the researchers found that across 28 translations the longest-option advantage over chance stays between about 94% and 107% of the English level. On biology and psychology questions the strategy still scores roughly six times chance in every language. The correct answer often remains the longest after translation, though which option is longest frequently changes. The authors conclude that benchmark scores should be compared against this length-only baseline, which is roughly 18% overall, rather than the 10% chance level.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Why Birthday Reminders Fall Short When They Only Flag the Date

A simple birthday reminder usually arrives too late for people who need time to buy gifts, book reservations, or mail something. The article argues that birthday apps should go beyond storing dates and support advance planning, since different people need different lead times. It proposes multiple reminder points, such as a month, a week, and a day before the event, to give users time to act. It also suggests storing useful context like relationships, gift ideas, and preferences in one place. The piece ends by promoting Annum, an Android birthday planner built around preparing for upcoming events.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Three-Layer Retry Plan Aims to Make Webhook Failure Alerts Safer for AI Rollbacks

The article argues that a failed webhook in a healthtech AI-agent loop is more than a delivery issue, since it can obscure latency breaches and delay rollback decisions. It recommends a bounded retry policy using exponential backoff, a maximum attempt count, and a circuit breaker that halts load on a failing destination. Each delivery should be tracked as a measured operation, with attempt number, timing, response class, and trace identifiers recorded in structured logs. A separate poller is suggested to surface persistent failures so the alert path does not itself become an incident. The piece also includes a Go example separating retry control from the delivery function and reading error data from an observation API.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Exporting Competitors' Google Ads from the Transparency Center With Python

Google's Ads Transparency Center lists the ads each advertiser runs, including text, images, video, regions and dates, but it has no export button or public API. The author built an Apify actor that pulls this data into CSV-ready rows, and the article walks through a short Python script using the apify-client library. Returned records include fields such as first and last shown dates, days running, format and a transparency link, with an optional per-domain summary row. The author suggests tracking the daysRunning field, since long-running ads may be performing well, and notes that the tool can run on a schedule to flag newly launched creatives. The article also cautions that the Transparency Center does not reveal spend, clicks or targeting, and that ad text is not always available.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

A2A Protocol Aims to Let Independently Built AI Agents Work Together

As AI applications grow more capable, developers face the challenge of getting independently built agents to communicate and collaborate. The article illustrates this with a GitLab merge request review, where separate security, testing, and architecture agents feed findings to a review agent. Agent2Agent (A2A), an open protocol originally developed by Google and contributed to the Linux Foundation, offers a standardized way for agents to discover each other, delegate tasks, and exchange messages. It works regardless of the framework, language, or vendor used, and does not require agents to reveal internal reasoning or tools. The protocol also supports progress streaming, artifact exchange, and authentication across agent boundaries.

0 comments Read more at DEV Community