Golden File Tests Are Failing AI Agents as Models Silently Drift

·1 views

Software testing teams relying on frozen 'golden file' snapshots are increasingly finding them inadequate for validating AI agents, as underlying models change without notice and outputs shift unpredictably. Unlike traditional software, AI agents produce tool traces, side effects, and variable prose rather than a single stable reply, making static string comparisons unreliable. A three-layer testing approach is proposed: contract checks for JSON schema and required tools, effects checks measuring real-world outcomes, and a live critic model that reviews agent traces for plausible but flawed reasoning. The critic layer is identified as the most commonly skipped, often deferred until a customer demo or incident, which allows undertested agents to reach production. The recommended workflow involves capturing redacted traces, running deterministic contract gates first, and only then invoking a cheaper pinned critic model on a clean remote host.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

ClickHouse Tiered Storage Cuts S3 Costs 71% While Keeping Query Speed Intact

Engineers running ClickHouse on AWS EC2 face steep storage bills, with 3 TB on gp3 EBS costing roughly $245 per month compared to about $67 on S3 at $0.023 per GB-month. ClickHouse's MergeTree engine stores data in immutable parts that can each reside on different storage backends, making native tiering possible without any application-level changes. A tiered storage policy keeps recent data on fast local EBS while automatically migrating older parts to S3 after a configurable period, such as 12 months, using TTL rules. A secondary trigger called move_factor pushes the oldest parts to S3 once local disk usage crosses roughly 80%, preventing hot storage from filling up. Crucially, existing SQL queries, drivers, and dashboards require no modification, as ClickHouse handles part location transparently at the storage layer.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer spends $20k and 40 billion tokens building open-source AI agent harness in Dart

A solo developer has spent approximately $20,000 across 40 billion AI tokens over three months to build >_Fa, an open-source agent harness written in Dart, with zero lines of code written by hand. The project was built using low-cost models such as Kimi K3 and GLM-5.3, routed per task to keep costs down. The resulting tool is a roughly 7 MB single binary that requires no runtime, installs in under a second, and runs across multiple platforms including mobile, web, CLI, and CI pipelines. Notably, the harness is self-building — an automated factory shipped five releases in a single day without any human involvement. The app has now received Apple App Store approval, making it possible to build mobile apps directly from a mobile device.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Browser vs. Node.js Event Loop: Key Differences Every JS Developer Should Know

JavaScript is single-threaded but handles high concurrency through the Event Loop, which delegates heavy tasks like I/O and timers to system threads while queuing callbacks for the main thread. Both browsers and Node.js use this mechanism, but their implementations differ significantly in structure and optimization. The browser Event Loop includes a rendering phase targeting ~60Hz frame updates, where microtask bloat can freeze the UI, and requestAnimationFrame is preferred for visual updates. Node.js splits its macrotask queue into distinct phases — Timers, Poll, and Check — and adds process.nextTick(), which executes before standard Promise microtasks and can starve the Event Loop if called recursively. Understanding these differences is critical for writing performant, non-blocking JavaScript in both environments.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

CSS Grid Layout: How grid-template-areas and grid-area Work Together

CSS offers powerful grid properties that simplify webpage layout design. The grid-template-areas property allows developers to define named sections within a grid container. Child elements can then be assigned to these named sections using the grid-area property. For example, setting grid-area: header on a child element places it within the section named 'header' in the parent grid. Together, these two properties make it easier to build structured, readable layouts without complex positioning logic.

0 comments Read more at DEV Community