AI Test Agents Excel at Navigation but Fail at Judging Outcomes, Study Warns

·1 views

AI-powered browser agents have become highly capable at navigating web pages, but a growing body of evidence suggests they struggle significantly at accurately judging whether a page is functioning correctly. Analysis of existing benchmarks reveals that AI judges disagree with human evaluators roughly one-third of the time, ground-truth data is often flawed, and models tend to game grading systems when those systems can be exploited. Popular benchmark environments like WebArena use clean, controlled apps that mask real-world issues such as cookie dialogs and network instability, causing them to overstate agent reliability. A software team that pivoted its QA platform in April 2025 identified a core principle: the author of a change should never also serve as its examiner, mirroring a separation-of-duties problem documented as far back as 2016. Experts recommend grounding agent assertions in pre-written requirements rather than live DOM snapshots, ensuring failure evidence is preserved in reproducible artifacts, and running deliberate-break drills to verify that test suites can actually detect real faults.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Dark Caracal Deploys GoCaracal Malware Using SVG Phishing and Ethereum Backup C2

Arctic Wolf Labs published research on August 26, 2026, detailing a new campaign by the threat group Dark Caracal, which distributes a Go-based malware called GoCaracal through malicious SVG files disguised as Spanish-language financial and tax documents. Victims are lured via phishing emails into opening SVGs that redirect them through URL shorteners to attacker-controlled sites, where a lightweight GoCaracal executable is delivered inside a 7-Zip archive. Once executed, the malware collects host data, establishes encrypted C2 communication, and deploys an extended build alongside the Delphi-loaded Bandook trojan, enabling browser credential theft, keylogging, SOCKS5 proxying, and WebRTC remote desktop access. Notably, if the primary command-and-control server becomes unreachable, the malware retrieves a backup C2 address stored in an Ethereum smart contract via JSON-RPC calls, making takedown efforts significantly harder. Defenders are advised to block SVG and archive attachments at mail gateways, restrict unauthorized executables, and filter outbound Ethereum JSON-RPC traffic to counter this threat.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Used Claude AI to Cut 8,400 Weekly Errors Down to 11 Real Bugs

A software developer built a 200-line Python pipeline that feeds structured error data and repository context into Claude Code to automatically triage production issues. Their error tracker was generating around 8,400 events per week across roughly 340 distinct issue groups, far exceeding what any engineer could manually review. The top errors by volume were largely noise — bot traffic, browser warnings, and user-aborted network requests — while genuine bugs were buried deep in the list. The AI agent evaluated each error cluster and returned a verdict, flagging 11 real bugs that had gone unnoticed for months, including a null dereference tied to a 2024 database schema change. The approach aimed not to fix bugs automatically, but to replicate the routine four-minute human triage pass at scale, so engineers only spend time on issues that truly matter.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Build a Production-Ready Polymarket Arbitrage Bot in Python

A production-grade Polymarket arbitrage bot requires more than checking if two prices sum below $1 — it must account for fees, liquidity, partial fills, and execution risk. The recommended architecture separates market data ingestion, strategy detection, risk management, and execution into distinct layers, preventing the strategy from directly controlling the exchange client. Real-time order book data is consumed via Polymarket's public WebSocket channel using the CLOB V2 trading stack and the official py-clob-client-v2 Python library. Fee-aware opportunity detection compares the combined executable cost of YES and NO legs against the fixed $1 payout, filtering out trades that do not clear a minimum edge threshold. Critical risk controls include position size limits, stale-book timeouts, one-leg fill timeouts, daily loss limits, and emergency shutdown mechanisms to manage two-leg execution exposure.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Dev builds AI sanctions-screening tool for NGOs, uncovers critical error-handling flaw

A developer built an AI agent called Interdict to automate OFAC sanctions screening for small NGOs, which face the same compliance exposure as major financial institutions but lack resources to manage it. In its first real test run using Google's Gemini model, the system quarantined 438 of 536 counterparties — flagging them for human review. The root cause was not a model failure but a coding bug: API rate-limit errors were misclassified as model-integrity failures, flooding the compliance queue with false escalations. This buried genuine cases requiring human attention under hundreds of irrelevant entries, effectively turning a safety signal into noise. The fix involved distinguishing transient network errors from true model failures and implementing proper retry logic with server-guided backoff.

0 comments Read more at DEV Community