AI-Generated Tests Scored 94% Coverage But Failed Basic Mutation Testing

·1 views

A developer used an AI model to generate a pytest test suite for a small Python CSV-parsing library, achieving 94% code coverage with 20 auto-generated tests. However, when mutation testing was applied by deliberately altering the source code logic, one mutation survived undetected — revealing a gap in the AI-written tests. The AI had written tests for missing 'Time' values but never tested the case where a row lacked a 'Course' field entirely, leaving that logic branch unverified. A second mutation, which changed the return value of an invalid date parse, was correctly caught because the AI had explicitly tested for a None return. The experiment highlights a key limitation: high code coverage measures which lines were executed, not whether the underlying logic is robustly verified against behavioral changes.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer builds local-first linter to flag vague or incomplete release notes

A developer has created ReleaseProof, a local-first linting tool designed to improve the quality of software release notes. The tool takes two inputs — change evidence such as PR titles or diff summaries, and a release note draft — then checks for vague claims, missing audience details, and absent migration information. It operates without requiring an account or repository access, and does not attempt to rewrite notes or verify code correctness. Deterministic rules underpin the checker, which has been validated against 20 focused test cases and five real-world release notes from projects including uv, Ruff, pnpm, and Bun. The developer is seeking user feedback on false positives, missed issues, and overall usefulness of the tool.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Server Infrastructure Powers Web Applications: A Technical Overview

Server infrastructure forms the backbone of web applications by handling user requests, processing data, and communicating with databases before returning responses. A typical request follows the path from the user through the internet to a web server, then a backend application, and finally a database. Virtual servers, managed by hypervisors, allow multiple environments to share a single physical machine, offering benefits such as cost efficiency, scalability, and easier backup and recovery. An operating system on the server manages critical functions including process handling, memory, networking, security, and logging. Effective server management also requires ongoing traffic monitoring, performance tracking, security hardening, and reliable backup procedures to maintain stable and secure application delivery.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Configure an IPv6-Only VM and Container Network Using Podman

Despite IPv6 being standardised roughly 28 years ago, many applications still rely solely on IPv4, prompting developers to test IPv6-only environments. A virtual machine can be switched to IPv6-only by first enabling IPv6 via sysctl, then reconfiguring network interfaces with nmcli before disabling IPv4 entirely. Before disabling IPv4, users must reconnect via an IPv6 SSH session to avoid losing access to the VM. For containerised workloads, Podman allows creation of an IPv6-only network by specifying an IPv6 subnet, using a private fd00::/8 address range analogous to IPv4's 192.168.x.x. A dual-stack option also exists in Podman, and its built-in DNS service supports both IPv4 and IPv6, enabling container-to-container name resolution.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Running LM Studio Locally Still Requires Network Isolation Controls

LM Studio and similar local LLM applications can still make external network connections for tasks like downloading models, fetching updates, and connecting to MCP servers, even when inference runs entirely on-device. A developer outlined a two-layer security approach combining LM Studio's built-in settings with OS-level network controls to limit external connectivity. The setup separates two distinct phases: an initial setup phase where network access is permitted for installation and downloads, and a hardened daily-use phase where external connections are blocked. At startup, a wrapper script unloads all existing models and loads only a single pre-approved model, ensuring the environment remains consistent and predictable. The author emphasizes treating local execution and network isolation as separate problems, rather than assuming a local LLM setup is automatically contained.

0 comments Read more at DEV Community