Same CLI Tool, Wildly Different Scores: Why Harness Names Mislead AI Benchmarks

·1 views

Specific Labs released Real-SWE, an enterprise-focused software engineering benchmark, revealing a critical flaw in how AI coding tool results are commonly reported. Two models run on the same Codex CLI harness scored 33.8% and 16.2% respectively, a gap of more than 2x despite sharing the same interface. Similarly, two models running on Claude Code differed by 10 percentage points, underscoring that the harness is merely a routing layer, not the reasoning engine. Real-SWE addresses this by framing every result as a model-and-harness combination, a standard most vendors avoid because low scores under their own tooling reflect poorly on them. Experts warn that teams evaluating coding agents should always identify both the underlying model and the harness, as the model swap is the single biggest performance lever yet remains largely invisible in vendor marketing.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Atlassian's AI PR Reviewer Cuts Cycle Time by Shifting, Not Eliminating, Review Work

Atlassian's Rovo Dev AI reviewer reduced pull request cycle times by up to 45% internally and 32% for customers by automating mechanical checks such as standards enforcement and Jira acceptance criteria before a human ever opens the PR. However, the time saved largely reflects faster routing and pre-screening, not a reduction in the core decision-making step. The Real-SWE benchmark found the best AI agents resolved only 38.8% of tasks on enterprise codebases, underscoring that human judgment on whether a change is correct remains essential. Teams that only accelerated the reading phase found their queues moved faster without reducing the rate of errors reaching review. The meaningful gains come from restructuring the process so machines handle all automatable checks first, leaving humans to focus solely on the correctness decision.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Giving AI Coding Agents More Time Does Not Improve Success Rates, Study Finds

A benchmark called Real-SWE tested leading AI coding agents against private enterprise codebases covering billing, tax, and multi-service workflows. The study found that extending agent runtime made virtually no difference — failure rates hovered around 71–73% regardless of whether a task ran under or over 10 minutes. The top-performing system, Fable 5.1 paired with Claude Code, resolved only 38.8% of tasks, while GPT-6 Astra on Codex CLI reached 33.8%. Researchers noted that agents tend to fail not due to insufficient compute time, but because of gaps in contextual understanding or inadequate scaffolding. The study also highlighted that benchmark scores reflect a specific model-plus-harness combination, warning that comparing results across different scaffolding setups leads to misleading conclusions.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Agent-Cache Brings Three-Tier LLM Caching to Valkey and Redis

A new open-source library called Agent-Cache addresses inefficiencies in AI agent loops by introducing a three-tier caching system backed by Valkey or Redis. The project caches LLM responses, tool call outputs, and session state separately, each with its own TTL strategy suited to how frequently that data changes. Released as v0.1.0 with support for Valkey 7+ and Redis 6.2+, it quickly followed with v0.2.0 adding cluster mode, and includes adapters for LangChain, LangGraph, and Vercel AI SDK. Cache keys are constructed from prompt hashes, model parameters, or function arguments, meaning any change in input automatically bypasses the cache and triggers a fresh API call. When the cache backend is unavailable, the library defaults to graceful degradation — continuing agent execution by hitting the LLM or tool directly — though this behavior can be configured per tier.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Uses Hackathon to Discover CDO Role Matching His Own Profile via RSS and AI

The HackAVP, described as New Caledonia's first hackathon dedicated to public sector job vacancies (AVPs) from OPT-NC, was launched this week. During a pitch session, a participant highlighted the availability of an RSS feed on the platform, noting its utility for quick, low-code integrations. As a practical demonstration on the SaaS track, he built a scheduled job within Claude CoWork to automatically find, match, and prepare job application documents stored directly on Google Drive. Without writing any code, the system identified vacancy 3134-26-1315_sr as the best match for his profile — which turned out to be a Chief Data Officer opportunity within his own organization. The demo was submitted via the hackathon's online form as a concrete example of what participants can build on the platform.

0 comments Read more at DEV Community