Why Your AI Agent's 'Cost Per Success' Metric Is Probably Wrong

·1 views

A software developer has highlighted a critical flaw in how AI agent task costs are typically measured on monitoring dashboards. Most systems count a task as successful when an agent self-reports completion via an exit code of zero or an HTTP 200 response, even if the actual intended outcome never occurred. This means silent failures — such as a rate-limited API returning a 200 status with no real result — are incorrectly logged as successes, artificially lowering the reported cost-per-success figure. The proposed fix is to replace self-reported success counts with independently verified outcomes, such as confirming a database row was written or a file was created. Dividing total agent spend by only these externally witnessed successes produces a 'cost per verified success' that more accurately reflects what was actually accomplished.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

AI writes the code, but the developer remains fully accountable for it

A software developer reflects on merging a pull request largely written by an AI coding agent, noting that his name — and therefore his accountability — remained on the work regardless of who typed it. He argues that AI assistance does not reduce ownership; it increases it, because the developer must supply the business context, system knowledge, and judgment that the model inherently lacks. The agent can produce syntactically correct code while missing critical details such as multi-tenancy checks or deprecated libraries known only through internal team history. Having a second AI review the output helps catch errors but does not create responsibility, since no model attends a postmortem or answers to a customer. The author concludes that as AI accelerates code output, the human role shifts toward higher-stakes judgment calls, and the number of people accountable when something breaks remains exactly one per pull request.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Builds Tiny Language Model in Pure Node.js With No ML Frameworks

A developer has built a minimal language model entirely in Node.js without using TensorFlow, PyTorch, or any external dependencies, making every step of the learning process visible in code. The project, called tiny-language-model-neuro-js, runs on Node.js 18.19+ with a single command and implements the full pipeline from tokenization to backpropagation using scalar-level operations. Before training, the model produces random wrong answers to simple questions; after pre-training, supervised fine-tuning, and adaptive SFT, it answers all test queries correctly. The acceptance criterion requires every target token to reach at least 95% probability and the full check to pass more than ten consecutive times. The project is designed as an educational tool, with the code structured to read as a single narrative rather than a framework, making each mathematical operation directly traceable in the source.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer builds browser tool to give AI coding agents verifiable UI checks

A developer has released hwatu, an open-source browser tool built to let AI coding agents verify UI changes with measurable precision instead of making unsubstantiated claims of success. The tool runs a full verification loop — including page load, DOM checks, screenshot capture, and pixel-level diffing — in approximately 87 milliseconds. Unlike Chromium-based solutions, hwatu is a single static Rust binary that relies on WebKitGTK 6.0, avoiding large per-project downloads. It supports three integration protocols — MCP, CLI, and a JSON socket — making it compatible with agents like Claude Code, Cursor, and Jcode. The project is MIT licensed and currently available for Linux, with source code and benchmarking methodology published on GitHub.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Terraform and YAML: Advanced Patterns for Enterprise-Scale Infrastructure

A technical article series on using Terraform with YAML for infrastructure-as-code has reached its third and final installment, focusing on enterprise-level patterns. The piece explains the limitations of Terraform's native shallow merge function, which only combines top-level keys and can cause nested configuration data to be lost. To address this, the article introduces deep merge strategies that allow more specific environment configurations to override generic global settings without discarding shared values. It also covers organizing multiple YAML files across different scopes such as global, environment, service, and region layers. Additionally, the article outlines how these configuration patterns can be integrated into CI/CD pipelines to automate infrastructure deployment workflows.

0 comments Read more at DEV Community