Benchmark built to verify AI claim found its own grading code was flawed

·1 views

A developer at DEV Community ran a structured benchmark to test a claim that a two-year-old coding-trained model outperforms a newer general-purpose model at structured-text tasks. Two Qwen3 models were evaluated across five task types — JSON repair, YAML frontmatter, CSV cleanup, Docker log extraction, and markdown tables — with each fixture run three times for reliability. Before results were recorded, a self-test mode revealed the YAML grader contained a bug where a date string was being compared to a Python datetime object, meaning correct answers would have been silently failed. The Docker log fixture showed the only performance difference between models, but this too turned out to stem from an ambiguous prompt rather than a genuine capability gap. The episode highlighted that evaluation harnesses demand the same adversarial scrutiny as the models they are meant to test.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Wrong AdMob App ID Caused Limited Ad Serving on Live Android Game

A developer discovered that AdMob was serving limited ads on their live Android game, Logo Quiz: Guess Brand, because the ad units were registered under a different AdMob app entry than the one linked to the Play Store listing. AdMob requires a one-to-one link between a Play Store listing and an AdMob app, and since the verified app had no ad units while the app with ad units was unverified, neither could serve ads properly. The fix involved creating ad units under the Play-linked app and updating four IDs across the project, including in both app.json and AndroidManifest.xml, where the app ID is stored in two separate places. The developer noted that changing only app.json still results in the old ID being baked into the APK, and recommends inspecting the compiled artifact directly to confirm which ID was actually shipped. The key lesson offered is to create and link the AdMob app to the Play Store listing before adding any ad unit IDs to the codebase.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Find the Real Cause Behind Unity's 'Gradle Build Failed' Error

Unity developers building Android APKs often get stuck on the 'CommandInvokationFailure: Gradle build failed' console message, which appears hundreds of lines after the actual root cause in the log file. The true error — such as duplicate classes on the classpath — is logged around line 923, while the misleading Gradle failure message appears near line 1158. Developers can bypass the noise by running a grep command against Unity's Editor log to jump directly to the 'What went wrong' section. Common triggers include duplicate JARs in the project, such as copying unity-classes.jar under a second name, or conflicts introduced via the External Dependency Manager. Clearing resolved libraries through Assets → External Dependency Manager → Android Resolver can resolve duplicate-class errors, particularly in Unity 2022.2 and later.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Dev Writer Tests Every Terminal Claim on His Machine Before Publishing

A developer and technical writer behind the weekly newsletter Shellcraft has shared his core editorial rule: every terminal-related claim is verified on his own machine before publication. He estimates that roughly one in five drafted claims turns out to be incorrect upon testing, not due to typos but plausible-sounding errors rooted in outdated or misremembered information. To enforce this standard, he creates a private verification guide for each issue, outlining steps to reproduce commands and their expected outputs. He recently caught a critical error while testing a tool's panel workflow, discovering that a central step in a drafted issue did not work as described, prompting a full rewrite. Introducing himself to the DEV Community, he framed this practice as a promise to the developer audience he plans to write for there at least once a month.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Solo Dev Builds Ad-Free HTML5 Gaming Platform With 17,000 Games and 500K Players

A solo developer built Spielcade, a free browser-based gaming platform designed to eliminate the ads, malware warnings, and forced downloads common on casual gaming sites. The platform hosts over 17,000 HTML5 games across more than 25 genres and has attracted over 500,000 active players with a 4.9 out of 5 rating. It was built using Next.js, React, Node.js, and PostgreSQL, with Cloudflare CDN used to ensure fast global load times. Key technical challenges included scaling database queries for complex tag filtering, optimizing thousands of game thumbnails via lazy loading, and managing technical SEO across more than 17,000 individual game pages. The developer used server-side rendering and automated dynamic sitemaps to help search engines index the growing library effectively.

0 comments Read more at DEV Community