Why You Must Freeze a Holdout Set Before Trusting Any Coding-Agent Score

·6 views

Coding-agent benchmarks can quietly become meaningless when teams repeatedly inspect failures, adjust system prompts, and rerun the same tasks — inflating pass rates without genuine improvement. A structured protocol addresses this by splitting a task suite into a tune set and a permanently frozen holdout set, with scores only published from the untouched holdout. Each task must meet strict validity rules: the unpatched repo must fail hidden tests, the oracle patch must pass them, and hidden tests must never be visible to the agent. The holdout split should be generated with a fixed random seed rather than human intuition, which tends to place harder tasks in the tune set and skew results. Even a small suite of twenty well-formed tasks is considered more reliable than hundreds of loosely defined demos, since a frozen denominator, meaningful metrics, and grader integrity checks are all required before a percentage qualifies as a real measurement.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer Experiment Reveals Hidden Latency After AI Code Generation Ends

A developer found that the real performance bottleneck in an AI-assisted coding workflow was not the remote language model but the file-writing steps that followed token generation. Standard product dashboards typically highlight first-token latency, masking heavier stalls that occur during the apply phase. To investigate, the developer built a lightweight Python timer using perf_counter to record named spans and render them as an ASCII waterfall chart. The experiment used a small local JSON fixture with no production traffic, deliberately keeping the setup simple and reproducible. The findings showed that apply-phase operations like writing and fsyncing files produced multiple distinct spans, collectively outweighing the generation span in wall-clock time.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Study Finds Half of 330 AI Models Fail to Respond Correctly in Korean

A benchmark evaluation of 330 large language models tested Korean language ability across seven axes, including honorifics, terminology, register, and format compliance. A script-based script-contamination check — not an AI judge — automatically failed any response containing less than 25% Hangul, more than three Chinese characters, or any Japanese kana. Using this method, 766 of 2,304 responses (33.2%) were auto-failed before human or AI review, and 54 models failed all seven tasks entirely. Honorifics and knowledge of Korean institutions proved the hardest axes, with AI success rates of just 8.5% and 9.4% respectively. Among vendors with at least eight models tested, Anthropic led with a mean score of 2.29 out of 3, while five models — mostly from OpenAI — achieved a perfect score.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Faster Software Development Means Little If Users Still Struggle to Navigate It

Modern tools like agile workflows, component libraries, cloud platforms, and AI-assisted coding have dramatically reduced the time it takes to build software. However, writer Parvej Shah argues that development speed and product usability are fundamentally different metrics, and shipping features quickly does not guarantee those features are easy to use. Shah contends that the true advantage of rapid development lies not in building more features faster, but in shortening the feedback loop — testing assumptions with real users sooner and iterating based on what is observed. While engineering teams measure progress through deployments and passing tests, users only care whether a product helps them accomplish their goal. As competitors also accelerate their development pace, Shah suggests the real strategic question is not how to build faster, but how to use that speed more wisely.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

React Router Data Mode Explained: How v6 Upgraded Single Page App Management

React Router offers three progressive modes — Declarative, Data, and Framework — each adding more features at the cost of architectural control. Declarative Mode handles basic URL-to-component matching using APIs like useNavigate and useLocation, while Data Mode, introduced in React Router v6, moves route configuration outside React rendering to enable data loading, actions, and pending states. Data Mode uses APIs such as loader, action, and useFetcher, giving developers more hands-on control including parallel data loading. Framework Mode builds further on Data Mode by integrating a Vite plugin to support type-safe routing, intelligent code splitting, and SSR or static rendering strategies. This article series focuses primarily on Data Mode, aiming to provide a deep-dive into its capabilities and practical usage.

0 comments Read more at DEV Community