Targeted AI Test Generation Caught 44 of 46 Killable Bugs, but With a Catch

·1 views

A developer ran an experiment testing how well AI-generated tests could detect deliberately planted bugs in Python code using a mutation-testing framework. A generic prompt producing 556 tests caught only 9 of 53 surviving mutations, while a targeted approach — writing one test per mutation with a strict pass/fail gate — caught 44 of 46 that were realistically detectable. However, the tests showed zero cross-function transfer, with 36 of the 44 catching exactly one mutation each, suggesting the AI learns the fault shown rather than improving broader coverage. The experiment also uncovered 11 bugs in the harness itself, each of which made results appear better than they were, highlighting how measurement errors tend to go unnoticed when outcomes look favorable. The author concludes that high test counts and line coverage metrics can be deeply misleading without mutation-based verification of what tests actually detect.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Anthropic Proposes Three-Metric Framework to Track AI R&D Automation Progress

Anthropic has outlined a more comprehensive approach to measuring advanced AI systems, arguing that benchmark scores alone fail to capture the full picture of AI-driven research progress. The framework, supported by an arXiv preprint titled 'Measuring AI R&D Automation,' identifies three key indicators: how well AI performs on research-like tasks, how extensively it influences high-stakes decisions, and whether its use introduces attempts to disrupt research processes. Anthropic's Claude Opus 4.5 System Card serves as the most detailed first-party record of automated AI R&D evaluations, including analysis of conditions under which AI might substitute for human researchers. The company's Transparency Hub and model reports provide additional context on autonomy assessments, capability evaluations, and deployment safeguards. Rather than a one-time product release, the effort reflects an ongoing measurement and transparency program as AI increasingly plays a role in developing and evaluating AI itself.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Manage Internal DNS Hostnames Reliably Using Infrastructure Code

A deployment workflow for managing stable internal DNS records involves storing them in an infrastructure repository, upserting records during deployment, and verifying published values match the repository's intent. The process uses a post-apply set comparison between the repository's desired state and live DNS answers to catch discrepancies such as out-of-band edits or incomplete rollbacks. Engineers are advised to choose a DNS provider based on existing control plane ownership — AWS Route 53, Cloudflare, or Google Cloud DNS — rather than convenience or familiarity. The workflow deliberately excludes frequently changing service endpoints, which are better handled by a dedicated service registry. A bounded polling approach is recommended during verification to account for DNS propagation delays rather than assuming instant convergence.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Backboard.io Adds Support for TypeSafe's Jev AI Decision Models

Backboard.io has announced support for TypeSafe's Jev, described as a viral AI model. The integration falls under Backboard's System One Models category. Jev support covers three types of structured decisions: Choice, Score, and Noul, the last of which handles true/false probability outputs. Documentation for the new functionality is available on the Backboard.io developer docs site.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

No Single AI Coding Framework Wins — Experts Recommend Building a Stack

A DEV Community analysis argues that the popular debate over which agentic coding methodology to use is the wrong question to ask. The article reviews several frameworks — including Superpowers, BMAD, OpenSpec, SpecKit, GSD, Ralph, and Compound Engineering — each designed to handle a distinct layer of AI-assisted development. Rather than seeking a universal solution, developers are advised to sequence these tools based on project context, such as using BMAD when product intent is unclear and Superpowers for rigorous implementation tasks. Compound Engineering is highlighted for founder-leaning roles, as it focuses on making each delivery easier for future AI agents by capturing reusable lessons. The author recommends starting with BMAD as a high-level harness combined with OpenSpec to maintain implementation quality.

0 comments Read more at DEV Community