AI models ace flaw detection but flag clean code as faulty, benchmark finds

·2 views

A developer built a 28-item benchmark called Blog vs Bytecode to test whether AI models evaluate data-science code or simply trust the accompanying blog-style description. The benchmark covers six flaw categories including data leakage, metric mismatch, and train-test contamination, using matched adversarial pairs to distinguish genuine understanding from pattern recognition. Top models such as Gemini 3.1 Pro, DeepSeek-R1, and Grok 4.20 with reasoning enabled achieved 100% accuracy, but several mid-tier models over-flagged correct code as problematic. Grok 4.20 dropped from 100% to 68% accuracy when reasoning was disabled, highlighting that code-reading capability depends heavily on the reasoning mode. The benchmark also exposed a data-capture flaw in Kaggle's model proxy that made strong models appear to score as low as 4%, skewing results until blank responses were excluded from grading.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

WordPress Click2Shell Flaw Exposes RCE Risk Across Millions of Sites

A vulnerability dubbed Click2Shell, disclosed on September 21, 2026, revealed an unauthenticated remote code execution chain in WordPress Core that tricks a logged-in administrator into visiting a crafted URL, which then silently installs a malicious theme and executes attacker-controlled PHP. WordPress addressed the flaw in version 7.1.1 via changeset 63664, though no CVE has been assigned and no active exploitation in the wild has been reported. A ZoomEye query run on September 22, 2026 returned nearly 7.95 million WordPress assets globally, though researchers caution this figure represents exposure scope, not confirmed compromised hosts. The attack has two layers: a core parser flaw affecting all versions before 7.1.1, and a secondary stage exploiting unprotected AJAX endpoints found in over 40 catalog themes, including Mobile Repair Zone 2.5.4. Because the chain depends on social engineering an administrator to click a link, site owners are urged to patch to 7.1.1 and audit installed themes rather than treat the asset count as a breach tally.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer builds script to verify all 8,556 PDF links in 871-page digital planner

A software developer created a Python-generated 871-page hyperlinked PDF planner containing 8,556 internal link annotations spanning daily, weekly, and monthly pages. To ensure quality, they wrote a verification script using pypdf that checks every link annotation, resolves its destination, and flags anything broken or out of bounds. Beyond basic link resolution, the script performs semantic checks — confirming, for example, that a calendar cell for March 14 actually lands on the correct page — and walks the full 105-week prev/next chain to catch year-boundary off-by-one errors. The shipped build passed with zero broken links out of 8,556 checked, though the script proved its value earlier in development by catching bugs caused by page insertions that silently remapped link destinations. The developer notes that on-device testing in apps like GoodNotes and Notability remains an outstanding gap not covered by the automated checks.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Missing description strings silently hide AI agent handlers from tool manifests

A development team discovered that their in-app AI agent could not access dozens of existing handlers because each lacked a required description string in its definition. Without that string, handlers are excluded from the manifest the agent receives, making them effectively invisible regardless of whether the underlying code works. Twenty-one handlers across the team's AI pipeline and prompt store features were affected, all functioning normally for human users but unreachable by the agent. The team also implemented a two-tier risk system, marking irreversible actions like hard deletes and free-text prompt edits as high-risk to prevent the agent from being granted permanent approval for those calls. Automated gap tests were added to each feature to catch undocumented handlers before they ship, since silent omissions produced no errors and were difficult to diagnose.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

DOGFOOD Platform Lets Anyone Recompute and Verify Hackathon Results Independently

A new open-source hackathon platform called DOGFOOD, built by Hackathon Raptors, aims to make judging results fully transparent and independently verifiable by anyone. The platform uses a weighted scoring rubric combined with a statistical bias-removal model to adjust for judge leniency, then ranks projects by bias-corrected quality scores. When results are published, a signed bundle containing raw scores, weights, methodology, and code is hashed and signed with an Ed25519 key, allowing a standalone Python script to recompute and verify the ranking. Any alteration to the data, such as changing a single judge's score, causes verification to fail immediately. The project is available under the MIT license on GitHub and includes a public demo showing the full workflow from event creation to tamper-detection.

0 comments Read more at DEV Community