SShortSingh.
Back to feed

METR's Updated Time Horizon Plot Shows AI Coding Gains Outpacing Judgment Tasks

0
·8 views

METR, an AI evaluation nonprofit, has published an updated time horizon plot measuring how long AI models can autonomously complete tasks before failing. For software engineering benchmarks, the plot shows the time horizon doubling roughly every seven months since 2021, growing from seconds to over an hour. The plot only tracks tasks with clear right answers, such as code that compiles or fails. METR notes that models lag on judgment-based tasks, whose time horizons are shorter and improving more slowly. The findings suggest coding automation will advance rapidly, while judgment-heavy work still requires human oversight.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Fourteen of Fifteen URLs Share One Page Title Due to a Next.js Directive

An analysis of a community website built with Next.js found that 14 of 15 measured URLs return the same generic page title. The cause is traced to the "use client" directive, which excludes components from the App Router's server-side metadata export. The author notes that this is a mechanism choice rather than a missing feature, since titles can be set in other ways. Measurements showed that client components are still prerendered and their text appears in the raw HTML. The only route that declares its own title is also the only one that does not use the directive.

0
ProgrammingDEV Community ·

Mathematician Claims AI-Assisted Proof of Riemann Hypothesis, Pending Expert Review

Anthropic mathematician Levent Alpöge stated on November 20 that he and Claude believe they have found a proof of the Riemann Hypothesis, a conjecture dating to 1859 that carries a million-dollar Clay Mathematics Institute prize. He says the proof covers the stronger Generalized Riemann Hypothesis and has been written in Lean, a language that checks each step of a mathematical argument. Terence Tao described it as a Lean-only proof and noted that verification and digestion by the community will take time. Experts still need to confirm that the hypothesis was correctly encoded in Lean, since a passing check only shows the proof matches the stated problem. The claim remains unverified while mathematicians review it.

0
ProgrammingDEV Community ·

Developer merges several PRs after AI-assisted work, then writes first fix by hand

A developer reports that three of their pull requests were merged this week into projects they don't own, after most of their earlier code had been written by Claude Code. The developer then hand-typed a four-line fix to a command-line runner that reported crashes with a success exit code, and explained the change before merging. Other merged contributions included a reviewer-requested update to escaping logic and a patch to Django settings helpers that reset values instead of restoring them. The developer also began reviewing contributions from outside collaborators and reports disclosing AI assistance in their PR descriptions. The post closes with advice to have AI explain its code and to verify its claims.

0
ProgrammingDEV Community ·

Presage SmartSpectra SDK Challenge Entry Explores Human Signal Integration

A submission to the Presage SmartSpectra SDK Challenge describes a project focused on integrating human signals into technology. The work examines how such signals can make digital systems more responsive and interactive. It emphasizes the potential of interpreting human behaviour through technological means. The article is a challenge entry, and it does not provide extensive technical detail or results beyond this overview.