Developer Builds Code-Repair Benchmark That Tests Beyond Prompt Examples

·32 views

A software developer created 'invariant-repair-suite', a Python benchmark with eight code-repair cases, while participating in the Kaggle Benchmarking Challenge. The benchmark is designed to test whether AI models can preserve code behavior not explicitly demonstrated in the prompt, targeting quiet bugs that pass visible examples but break production invariants. Each case provides a contract, a buggy implementation, and two visible examples, while scoring weights hidden edge cases at 75% and visible regression checks at 25%. Reference fixes scored a perfect 1.0 overall, while the buggy starter implementations scored 0.57, confirming the harness can distinguish correct from plausible-but-incomplete code. The author outlines three core design rules: keep hidden test conditions out of the prompt, penalize regressions, and prefer executable assertions over model-based judgment.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Chinese developer seeks legitimate way to keep paying for Claude amid regional access limits

A 22-year-old developer in mainland China says Claude, especially Claude Code, is central to his coding and open-source work, including Minecraft mods. He has worked around networking barriers but still struggles to pay for a subscription because of regional restrictions. He acknowledges Anthropic's compliance and security reasons for those limits and says he is not asking anyone to bypass them. He is looking for developers in supported regions who might know a lawful arrangement, offering to cover costs and share server resources in return, while recognizing the privacy and trust concerns involved. He also invites advice and simply wants to connect with fellow developers.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Image File Signatures Like PNG, JPEG, WebP and AVIF Identify Formats

The article argues that a file's actual byte content is more reliable than its extension or declared MIME type for identifying an image format. Every common image container starts with a fixed byte pattern at a known offset, such as FF D8 FF for JPEG or 89 50 4E 47 for PNG. PNG's signature includes CRLF and DOS end-of-file bytes so that text-mode transfers corrupt it visibly. AVIF, HEIC and HEIF share the same ISO base media container and are told apart only by the brand code at offset 8. The piece also shows a short JavaScript check that identifies common formats from the first twelve decoded bytes, while noting that a signature confirms only the container type, not that the file is complete.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Checklist for Hardening AI Agents Before 2027 Launches

This one-page template helps teams define an AI agent project before building begins. It asks them to describe the task in a sentence, name business and technical owners plus a week-one reviewer, and classify each action by risk level. It also calls for listing data sources and sensitive fields, documenting where the agent runs, and recording baseline metrics with saved test cases. Finally, it requires a clear pause threshold, someone authorized to pause, and a date for the first review.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

A Personal AI Agent Built as a Folder of Markdown Files

A developer describes a personal AI agent that consists simply of a folder of Markdown files worked on by a coding agent, with no dedicated app or server. The system handles tasks such as workout planning, nutrition and protein tracking, spending logs against a budget, and voice-input journaling. The folder is organized into daily records, a Luhmann-style knowledge wiki with source links, a profile about the owner, and per-person files to avoid transcription errors with names. Additional modules such as fitness, writing and gaming are added as needed, and the folder can be opened directly in Obsidian for reading.

0 comments Read more at DEV Community