Study of 176 Agent Configs Shows Benchmark Numbers Miss What Buyers Need

·1 views

A nine-author study by Fan et al. tested coding agents across 176 configurations, isolating three core components — planning, action space, and context management — while holding four models constant. The research found that context management had the greatest impact on agent performance, while the role of planning varied depending on model strength. The findings highlight that popular benchmarks like SWE-Bench measure peak performance under ideal conditions, not real-world reliability or cost-efficiency. Experts and practitioners argue buyers should instead evaluate agents on their own tasks, tracking cost per accepted change rather than relying on vendor-reported scores. The study equips buyers to ask targeted questions about harness design, failure behavior, and per-component performance before committing to a tool.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

ClickHouse Tiered Storage Cuts S3 Costs 71% While Keeping Query Speed Intact

Engineers running ClickHouse on AWS EC2 face steep storage bills, with 3 TB on gp3 EBS costing roughly $245 per month compared to about $67 on S3 at $0.023 per GB-month. ClickHouse's MergeTree engine stores data in immutable parts that can each reside on different storage backends, making native tiering possible without any application-level changes. A tiered storage policy keeps recent data on fast local EBS while automatically migrating older parts to S3 after a configurable period, such as 12 months, using TTL rules. A secondary trigger called move_factor pushes the oldest parts to S3 once local disk usage crosses roughly 80%, preventing hot storage from filling up. Crucially, existing SQL queries, drivers, and dashboards require no modification, as ClickHouse handles part location transparently at the storage layer.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer spends $20k and 40 billion tokens building open-source AI agent harness in Dart

A solo developer has spent approximately $20,000 across 40 billion AI tokens over three months to build >_Fa, an open-source agent harness written in Dart, with zero lines of code written by hand. The project was built using low-cost models such as Kimi K3 and GLM-5.3, routed per task to keep costs down. The resulting tool is a roughly 7 MB single binary that requires no runtime, installs in under a second, and runs across multiple platforms including mobile, web, CLI, and CI pipelines. Notably, the harness is self-building — an automated factory shipped five releases in a single day without any human involvement. The app has now received Apple App Store approval, making it possible to build mobile apps directly from a mobile device.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Browser vs. Node.js Event Loop: Key Differences Every JS Developer Should Know

JavaScript is single-threaded but handles high concurrency through the Event Loop, which delegates heavy tasks like I/O and timers to system threads while queuing callbacks for the main thread. Both browsers and Node.js use this mechanism, but their implementations differ significantly in structure and optimization. The browser Event Loop includes a rendering phase targeting ~60Hz frame updates, where microtask bloat can freeze the UI, and requestAnimationFrame is preferred for visual updates. Node.js splits its macrotask queue into distinct phases — Timers, Poll, and Check — and adds process.nextTick(), which executes before standard Promise microtasks and can starve the Event Loop if called recursively. Understanding these differences is critical for writing performant, non-blocking JavaScript in both environments.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

CSS Grid Layout: How grid-template-areas and grid-area Work Together

CSS offers powerful grid properties that simplify webpage layout design. The grid-template-areas property allows developers to define named sections within a grid container. Child elements can then be assigned to these named sections using the grid-area property. For example, setting grid-area: header on a child element places it within the section named 'header' in the parent grid. Together, these two properties make it easier to build structured, readable layouts without complex positioning logic.

0 comments Read more at DEV Community