SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

AI-Written Tests Can Pass While Missing Critical Bugs, Mutation Testing Reveals

0
·1 views

A developer merged an AI-generated 14-test suite for an ISO date parser without review, only to discover the tests failed to catch broken new timestamp formats added a week later. The tests repeatedly validated the same happy-path scenarios with minor input variations, leaving key edge cases completely uncovered. Experts warn that a fully green AI-generated test suite does not guarantee meaningful coverage, and that false confidence from such suites can be more harmful than having no tests at all. Mutation testing — which introduces deliberate code bugs to check whether tests detect them — offers a measurable way to assess suite quality, with a score below 50% indicating largely decorative tests. A workflow using the open-source MonkeyCode platform, which currently offers free server access and a 10-million-token allowance, is proposed to automate this audit on every agent-generated suite.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingHacker News ·

Claude Code Plugin Unlocks Export-Blocked Kindle Highlights

A developer has published a Claude Code plugin on GitHub designed to recover Kindle highlights that are blocked from standard export. The tool, shared under the repository 'claude-plugins' by user l3a0, addresses a common frustration among Kindle users who cannot easily extract their annotations. The plugin leverages Claude Code's skill system to work around the export restrictions imposed on certain Kindle content. The project was shared on Hacker News, where it received minimal engagement at the time of reporting.

0 comments Read more at Hacker News
0
ProgrammingDEV Community ·

How a Per-Project Token Quota System Stopped One Batch Job From Starving Others

Three projects sharing a single LLM gateway and a 10-million-token allowance ran into trouble on day nineteen when a batch summarization job consumed 7.1 million tokens in four hours, causing all other projects to fail with quota errors. The root cause was a shared token pool with no per-project attribution, meaning any single consumer could exhaust the entire allowance unchecked. To fix this, a per-project quota layer was built in Python, assigning each project its own daily token budget and per-minute request limit, with all limits summing to 9 million to preserve a 10 percent safety buffer. The system runs a pre-check before each upstream request using a conservative token estimate, rejecting over-budget requests with a 429 response, then reconciles actual usage from the API response afterward. This two-step approach ensures that a single bad estimate causes only a minor overshoot rather than a runaway consumption event.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Developer Builds UrbanLease Appliance Rental Platform, Shares Key Technical Lessons

A developer built UrbanLease, a web platform allowing users to rent home appliances with features including rental history, payments, and separate user and vendor portals. Early in development, the project was migrated from SQLite to MySQL to better handle growing data complexity, requiring fixes to tables, columns, and database relationships. Feeling the initial version lacked differentiation, the developer added a rule-based smart recommendation system that filters and ranks appliances based on user inputs like budget and rental duration. A second feature, an appliance condition history log, was introduced to show prospective renters details such as past rentals, maintenance records, and service history. The developer noted that building a functional product is only part of the process, emphasizing that meaningful features should solve real problems rather than simply add more pages.

0 comments Read more at DEV Community
AI-Written Tests Can Pass While Missing Critical Bugs, Mutation Testing Reveals · ShortSingh