AI models tested on ability to predict data loss from destructive commands.

·3 views

A recent Kaggle benchmark tested 10 AI models on their ability to predict data loss from destructive filesystem, git, and SQLite commands. The models were given matched pairs of commands and a restore script, then asked to identify which data resources would be lost and recoverable. Performance varied widely, with Gemini 3.7 Flash scoring perfectly and some models scoring as low as 1 out of 18. The benchmark revealed that SQLite scenarios were the most difficult for the models, with a median accuracy of 58%. A significant finding was that 25 model answers incorrectly claimed no permanent loss would occur when the provided restore script would have left data unrestored.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Prevent lost test results by documenting exit codes before ephemeral hosts expire.

A development article advises teams to preserve crucial test data before ephemeral hosts are recycled. It warns that logs, commands, and exit codes can become permanently lost if not captured promptly. The article proposes a clear workflow involving designated roles of runner, reviewer, and reclaimer. It stresses that documentation should be stored in a wiki or repository, not in chat logs or on the temporary host itself.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Syed Jawad's Kaggle challenge entry tests AI models on data recovery claims.

Syed Jawad submitted an entry to a Kaggle benchmarking challenge on October 10. The project involved testing ten different AI models. The core question examined was their capability to restore deleted data. The work was shared as a community post on DEV Community.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Beginner developer uses AI agents to build and submit mobile game to four stores

A software development beginner in Korea directed multiple AI coding agents to create a mobile game over a 17-day period. The resulting game, 'Afterclose: Night Delivery', was submitted to four digital storefronts, including Google Play and the App Store. The developer's role involved high-level direction, testing on physical devices, and setting guardrails for the AI agents, rather than writing code directly. Despite over 1,450 automated tests passing, a critical bug affecting controls and sound was only discovered during manual testing on a real phone. The process involved splitting development tasks by file area across separate worktrees to manage parallel AI work.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Coding-Agent Evaluation Needs Negative Controls to Prevent Misleading Scores

An article advises developers to incorporate a negative-control slice when evaluating AI coding agents. This method prevents inflated scores from hidden test leaks, copied tasks, or broken assertions. The author recommends structuring evaluations into three distinct data slices: solvable tasks, negative controls, and canary tasks. Each task should be defined in a JSON object before testing to ensure methodological rigor. The goal is to maintain evaluation integrity and prevent unverified scores from being used for promotional purposes.

0 comments Read more at DEV Community