Why AI-Generated Tests Can't Verify Their Own Code Changes

·4 views

Software engineers face a hidden flaw in AI-assisted development: when an AI agent writes both a code patch and its accompanying tests in the same session, the tests cannot serve as independent verification. Because both artifacts share the same context and assumptions, a passing test suite may simply confirm the agent's own interpretation rather than the original ticket's requirements. The author proposes a three-lane workflow to address this, involving spec-isolated properties, content-addressed fixtures, and a skip budget that agent-generated diffs cannot modify. A key safeguard is deriving test contracts directly from ticket text in a separate workspace, before the agent session begins, with oracle files locked from agent edits. The article presents this as a practical workflow pattern rather than a formally studied solution, offering reproducible code skeletons for teams to adapt.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Blackwell GPU Inference Failures Traced to Backend, Runtime, and Memory Config Gaps

Developers testing large language models on NVIDIA Blackwell GPUs (sm_120 and sm_121) have documented how a single configuration change — such as switching MoE backends, adjusting memory utilization, or updating a runtime commit — can dramatically alter model behavior, available KV cache, or system stability. In one case, changing the MoE backend from flashinfer_b12x to marlin on a 96 GB RTX PRO 6000 freed 11.47 GiB of additional KV cache, while the b12x path's memory reservation caused outright failures on 16 GB and 32 GB Blackwell cards. A separate test found that non-deterministic logprob outputs in a 30B MoE model were resolved by disabling fused finalize, and an SGLang regression caused degenerate text generation until a specific upstream commit was applied. On a GB10 DGX Spark, setting GPU memory utilization to 0.92 instead of 0.60 led to an OOM condition invisible to the kernel, ultimately requiring a full power cycle after seven hours. A diagnostic tool called blackwell-doctor has been released to help users log the exact configuration fields needed to reproduce and report such issues accurately.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

64% of Unattended Claude Code Agent Failures Traced to Git Push, Not AI Errors

A developer team running 13 scheduled GitHub Actions workflows without human oversight logged 72 job failures since June, finding that 46 of them occurred at the git push stage rather than during model inference or API calls. Only 2 failures were attributed to the core task each agent was designed to perform, challenging the common assumption that AI decision-making is the primary risk in automated pipelines. One high-frequency polling job accounted for 60 of the 72 failures, largely because it ran often enough to collide with concurrent write operations. A concrete example involved a duplicate article being published after an idempotency key existed only in an uncommitted local file for 99 minutes, leaving the remote repository unaware the task had already been completed. The team concluded that engineering effort in unattended agent setups is better directed at write serialization and state consistency than at prompt robustness or tool-call error handling.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Defeats Rotating Bot Fleet by Exploiting Unforgeable Browser Header Pairs

In early September, a developer running an IP-reputation site detected tens of thousands of single-query requests per day, consistent with automated validation of a residential proxy pool across European ISPs. Simple per-IP rate limits proved ineffective since each bot exit node queried only once, exhausting third-party API allowances within hours. The operator's initial response of blocking specific HTTP header signatures failed repeatedly, as the bot fleet adapted its headers three times within 48 hours, using each blocked request as feedback to refine its impersonation. Attempts to flag infrastructure-level headers like connection keep-alive or forwarded IP fields were abandoned after testing revealed they also appeared on legitimate human traffic. The developer ultimately found reliable detection in logically inconsistent header combinations — such as a User-Agent claiming Windows while the client hint reported a different OS — which browsers generate consistently from a single source and cannot be faked without contradiction.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why Markets Can Be Calm and Terrified at the Same Time

On September 4, 2026, the VIX stood at just 14.53, placing it in the 18th percentile of its five-year range, signaling low near-term volatility. Yet the SKEW index simultaneously registered 151.58, near its 83rd percentile, indicating that options markets were charging near-record premiums for crash protection. SKEW, a Cboe index, measures how much more investors pay for downside put options versus upside calls, with readings above 100 reflecting heightened fear of a severe market drop. Analysts attribute persistently elevated SKEW levels to structural demand from pension funds and institutional hedgers, as well as a lasting market memory of sudden crashes like Black Monday in 1987. The divergence between VIX and SKEW illustrates that implied options pricing often captures tail-risk fears that recent realized volatility data alone cannot reflect.

0 comments Read more at DEV Community