SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

AI models lean toward longest wrong answers on MMLU-Pro's STEM-website questions

0
·10 views

An analysis of 15 models' published MMLU-Pro predictions found that when they answer incorrectly, they select the longest wrong option more often than chance would predict. The effect is strongest on the STEM-website biology and psychology questions, where it is larger than on questions derived from MMLU. Three weaker models scored below a simple longest-option heuristic that ignores the question entirely on those 656 questions. Even the strongest models showed the bias on STEM-website items, though their error counts were small. The researchers noted the lean is most visible where answer options are prose rather than numeric, since numeric options often tie in length.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Using GBase 8s onstat Diagnostics to Guide Evidence-Based Performance Tuning

A DEV Community article argues that database performance tuning should start from measurable evidence rather than observed symptoms. For GBase 8s environments, it describes using onstat commands such as onstat -g ses, onstat -g sql, and onstat -g act to inspect sessions, SQL activity, and workload behavior, with command availability to be confirmed per release. The piece recommends capturing baselines during normal and degraded periods, mapping symptoms like slow queries, blocking, or timeouts to relevant evidence, and avoiding high CPU or session counts as root causes on their own. It also suggests recording metrics such as response time, throughput, and storage latency, and following a cycle of hypothesis, single-factor change, and measurement, ideally in a representative test environment.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

NVIDIA Research Argues Small Language Models Suit Most AI Agent Tasks

NVIDIA Research published a position paper arguing that models under 10 billion parameters are sufficient, better suited, and far cheaper for most calls made by AI agents. The paper claims serving a 7-billion-parameter model can be 10 to 30 times cheaper than a 70-to-175-billion-parameter model, and that specializing a small model takes only a few GPU-hours. An independent 2026 DFKI study found the largest model tested used about 32 times the energy of a small one for roughly a 2-point accuracy gain on a math benchmark. The same study showed smaller models sometimes consumed more energy overall because they produced longer outputs in a substantial share of comparisons. Analyst data cited in the article indicate infrastructure limits are a major barrier to AI deployment for many organizations.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Three reusable developer prompts for PR review, changelogs and test scaffolding

A reusable prompt can be pasted into a fresh chat with new material and still return useful output without edits. The article argues this depends on settling one key decision in advance and writing it into the prompt. It offers three examples covering common sprint tasks: pull request review, turning commits into changelogs, and scaffolding pytest tests. Each example is tied to a different decision, such as a defined severity scale for reviews or a named target reader for changelogs. Naming the reader, the article notes, lets one line change produce a separate maintainer-focused version of the same prompt.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Styx wires seven coding agent CLIs into one app via four integration methods

The open-source Styx app runs seven coding agent command-line tools plus a plain shell on the same projects. Its developers found there is no single way to control an agent from another program, so they use four approaches: streaming JSON, a dedicated app server, the Agent Client Protocol, and an embedded terminal. Each method has trade-offs, such as whether mid-turn messages can be sent or whether approvals can be answered from outside the agent's own interface. A small router selects the right method per agent, so the rest of the app sees a uniform session type. The team also tests with fake CLI binaries to avoid contacting real providers or accounts.

0 comments Read more at DEV Community