SShortSingh.
Back to feed

Same CLI Tool, Wildly Different Scores: Why Harness Names Mislead AI Benchmarks

0
·55 views

Specific Labs released Real-SWE, an enterprise-focused software engineering benchmark, revealing a critical flaw in how AI coding tool results are commonly reported. Two models run on the same Codex CLI harness scored 33.8% and 16.2% respectively, a gap of more than 2x despite sharing the same interface. Similarly, two models running on Claude Code differed by 10 percentage points, underscoring that the harness is merely a routing layer, not the reasoning engine. Real-SWE addresses this by framing every result as a model-and-harness combination, a standard most vendors avoid because low scores under their own tooling reflect poorly on them. Experts warn that teams evaluating coding agents should always identify both the underlying model and the harness, as the model swap is the single biggest performance lever yet remains largely invisible in vendor marketing.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Tailnet-Gated Token Quotas for Small-Team LLM Gateways

A practitioner write-up describes how small engineering teams can stop runaway LLM spending caused by shared API keys copied across devices. The proposed design routes all model calls through a single egress proxy on a private Tailscale network, reachable only by team devices. A per-seat daily token cap is enforced before requests leave the network, using a rough character-based estimate rather than a full tokenizer. The article includes an access-control policy, a FastAPI gateway example, and an identity-mapping approach using Tailscale login headers. It also flags a race condition in the in-memory budget tracker that can let concurrent requests exceed the cap.

0
ProgrammingDEV Community ·

Knowledge-Centric Delivery: cómo los agentes de IA desplazan el cuello de botella de Agile

Un artículo de Rootstrap, presentado por el CTO Anthony Figueroa, sostiene que los agentes de IA han abaratado la implementación de software. Según esta tesis, el recurso escaso ya no es la hora de programación sino el conocimiento compartido, el contexto y la verificación. Figueroa propone Knowledge-Centric Delivery, un modelo en el que el conocimiento es la fuente y el código un artefacto derivado de él. El análisis del autor sostiene que no se trata de abandonar Agile, sino de completar sus valores, mientras que los rituales diseñados para racionar el esfuerzo pierden sentido. El texto también advierte que la revisión e integración de lo producido por los agentes siguen teniendo costo, por lo que el cuello de botella podría simplemente trasladarse.

0
ProgrammingDEV Community ·

Seven causes of GitHub Actions jobs stuck in Queued, and how to fix each

Jobs that sit in Queued for more than a minute or two are usually held up by one of seven specific causes, and adding runners only resolves some of them. The article explains that the GitHub API, not the Actions UI, exposes queue time through each job's created_at and started_at timestamps, and provides a command to list the longest waits across recent runs. It notes that a median wait measured in minutes signals a problem, while a healthy example showed a median of 13 seconds. The main culprit discussed is GitHub-hosted runner concurrency caps, which differ by plan, with macOS jobs limited to five concurrent slots on Free, Pro and Team plans. Suggested remedies include cancelling superseded runs, trimming test matrices, merging very short jobs, requesting higher limits from GitHub Support, and moving heavy jobs to self-hosted or third-party runners.

0
ProgrammingDEV Community ·

OpenJDK patches for Java4Shell (CVE-2026-114938) reach RHEL 8, 9 and 10

Fixed OpenJDK packages addressing the Java4Shell vulnerability (CVE-2026-114938) became available in Red Hat Enterprise Linux 8, 9 and 10 repositories on the night of November 18 and early November 19, 2026. Red Hat's security data API still listed the affected versions the morning of the 19th, even though the hosts' package manager already detected the updates. The corrected builds differ by Java line and RHEL version, and are tied to specific RHSA advisories that administrators should compare against installed builds. Rocky Linux and AlmaLinux were still awaiting rebuilt packages and mirror propagation at the time of the check. The advisory also covers eight other OpenJDK vulnerabilities and was released outside the regular quarterly cycle, so it may require separate change-management review.