Developer Builds Code-Repair Benchmark That Tests Beyond Prompt Examples

·3 views

A software developer created 'invariant-repair-suite', a Python benchmark with eight code-repair cases, while participating in the Kaggle Benchmarking Challenge. The benchmark is designed to test whether AI models can preserve code behavior not explicitly demonstrated in the prompt, targeting quiet bugs that pass visible examples but break production invariants. Each case provides a contract, a buggy implementation, and two visible examples, while scoring weights hidden edge cases at 75% and visible regression checks at 25%. Reference fixes scored a perfect 1.0 overall, while the buggy starter implementations scored 0.57, confirming the harness can distinguish correct from plausible-but-incomplete code. The author outlines three core design rules: keep hidden test conditions out of the prompt, penalize regressions, and prefer executable assertions over model-based judgment.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer Ditches Paid Snippet App, Rebuilds It Locally With Just 6 Commands

A developer abandoned a paid cloud-based code snippet manager after concluding that its sync convenience did not justify its cost or trust model. The core problem it solved — scattered snippets across bookmarks, notes apps, and scratch files — is real and measurable, costing developers time each time they re-derive code they have already written. The author argues that for working developers, the terminal is the most natural place to capture and retrieve snippets, making a local, CLI-based tool more practical than browser or cloud alternatives. A self-built snippet manager using six commands offers local-first storage, zero dependencies, and compatibility with version control. The author advises that such a tool only earns its place once a developer's snippet collection outgrows their editor's built-in system, typically signalled by repeated failure to locate code they know they once wrote.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

A single unparseable message silently halted shipment dispatches for 4.5 hours

A logistics platform lost nearly 4.75 hours of shipment dispatches one Saturday morning after a single malformed message caused a Kafka consumer to loop indefinitely on one partition. The problem began at 4:12 AM but went undetected until a depot called at 9 AM asking about a missing manifest, by which point 9,400 events had backed up. The root cause was an upstream team quietly changing a weight field to accept free text, allowing an operator override value of 'N/A' that the deserialiser could not handle. Rather than skip or quarantine the record, the consumer caught the exception, logged a warning, and retried the same message repeatedly for hours, generating 14,000 identical log lines that no alert monitored. The team subsequently introduced a dead-letter topic for unrecoverable errors, per-partition lag alerting on the oldest uncommitted offset, and a replay tool to reprocess fixed records.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Visa Raises Card Retry Limit to 20 in 30 Days, Catching Many Merchants Off Guard

Visa updated its Core Rules on 18 April 2026 to allow merchants up to 20 retry attempts on soft-declined cards within 30 days, raising the previous limit of 15. Major payment processor Adyen's documentation still references the old figure, suggesting many merchants may be operating on outdated retry logic. A card decline carries three distinct signals — an ISO 8583 response code, a scheme-level retry category, and a PSP-normalised code — and retry rules are tied to the first two, not the third. Visa classifies declines into four categories, with Category 1 codes barring retries entirely and Categories 2, 3, and 4 each permitting up to 20 attempts in 30 days. Adding complexity, the same response code can carry different meanings across networks — for example, code 65 signals a withdrawal frequency limit on Visa but triggers a strong customer authentication requirement on Mastercard in the UK and EEA.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Long Build Times Are Quietly Stealing Developer Focus and Productivity

Software developer Serguey Asael Shinder argues that slow build times have a hidden cost beyond the seconds they consume. When developers wait for builds to finish, they habitually fill the gap with distractions like email and chat, fragmenting their concentration. Repeated dozens of times a day, these micro-interruptions compound into significant lost focus. Shinder urges developers to measure their actual build loop duration and treat excessive wait times as a defect worth fixing. Where waits are unavoidable, he recommends intentional breaks rather than reflexive context-switching.

0 comments Read more at DEV Community