Why Comparing AI Agent Pass Rates Without Token Budgets Is Misleading

·18 views

A software engineering analysis argues that pass rates alone are an unreliable benchmark for evaluating AI coding agents when the token and tool-call budgets differ between runs. An agent that retries repeatedly until hitting a cap may outperform one given a single attempt, yet both results get averaged as if conditions were equal. The proposed protocol requires declaring a fixed budget before any model call, logging every attempt in a JSONL ledger, and publishing three metrics together: pass rate, truncation rate, and tokens-per-success. Task files must be hashed with SHA-256 so any fixture change invalidates the run ID, ensuring reproducibility. The author frames this as a methodological standard rather than a vendor comparison, noting that omitting budget data turns an engineering review into what they call 'a vibe with a percentage sign.'

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Geometric Deep Learning Extends AI Beyond Flat Grids to Graphs and 3D Structures

Deep learning has long relied on flat grids, such as image pixels and text sequences, which fit CNNs and Transformers well. Many real-world systems, including molecules, social networks, protein structures, and 3D point clouds, are irregular and non-Euclidean, so forcing them into grids can discard important spatial and topological information. Geometric Deep Learning addresses this by building neural operations around mathematical symmetries and geometric transformations, using Graph Neural Networks for relational data and manifold or equivariant networks for curved and 3D spatial domains. Applications highlighted include autonomous driving perception from LiDAR point clouds and detecting multi-account money laundering in financial transaction graphs. Proponents argue the approach offers the mathematical foundation needed for AI in physical sciences and spatial computing.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Running a Bilingual City Guide on Two Subdomains: Practical Lessons

Danang Plus is an independent guide to Da Nang, Vietnam, offered in Vietnamese and English on separate subdomains. The author explains that subdomains suited the project because the editions are written independently rather than machine-translated and can be deployed separately, though each must be verified and submitted to search tools on its own. Writing distinct content for each audience proved valuable, since English readers need background on matters like deposits and residence registration that locals already understand. The guide also recommends showing a "last checked" date beside volatile details such as prices and opening hours, and structuring sitemaps and canonical URLs carefully. The author concludes that writing for each readership and displaying update dates mattered more than any single technical measure.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How JavaScript's using statement disposes resources after returns and exceptions

The using declaration registers an object with a disposal method and calls it when the enclosing scope ends. In a Node.js 24.15.0 test, multiple resources were disposed in reverse order of acquisition, including after an early return. Failure cases showed that if a disposal throws, remaining disposals still run and errors are chained through SuppressedError, so logging only the outer message can hide the original failure. Acquisition failures do not trigger disposal of the resource being acquired, so the acquisition function must clean up partial work itself. Asynchronous disposal with await using behaved in order, and passing a value with only asyncDispose to using without await raised a TypeError.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Postman update allegedly installed plugins and a trusted hook into Claude Code and Codex

A developer reported in a GitHub issue filed on 25 October 2026 that upgrading Postman to version 12.33.4 silently installed Postman plugins into Claude Code and Codex. The reporter's installer log attributes the installs to the upgrade path itself, and their config files show a new plugin marketplace entry and a trusted record for a Postman SessionStart hook. That hook reportedly directs the agent to load Postman's API engineering skill before API tasks. The claims have not been independently reproduced publicly, and the original report does not document any sharing of API data.

0 comments Read more at DEV Community