SShortSingh.
Back to feed

Do AI-Written Tests Actually Verify Anything? A Project Tackles the Question

0
·2 views

A developer argues that tests generated alongside AI-written code can pass and turn CI green while verifying little about the implementation. The article illustrates this with a trivial always-true test and a discount fixture where a boundary bug at exactly 1,000 goes undetected by existing tests. To address this, the author built Veritas, an open-source project that examines which specification a test derives from and what it actually checks. Veritas uses mutation testing via mizchi/mutator to check whether tests detect code changes, rather than relying on the AI's own judgment. The author cautions that surviving or killed mutants do not by themselves prove correctness against the specification.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Test-Time Training Lets AI Models Update Their Weights During Inference

Modern deep learning typically freezes model weights after pre-training, so deployed models can only adapt to new information through their context window, and that learning disappears when the session ends. Standard recurrent and state-space models also compress past tokens into a fixed-size vector, which causes fine-grained detail to be lost over long inputs. Test-Time Training (TTT) addresses this by performing a self-supervised gradient update on fast internal weights each time a new token arrives during inference. Slow outer weights retain core reasoning and language ability, while the fast weights adapt to the current context and reset when the task changes. The approach is presented as a route to continual adaptation for AI systems facing unfamiliar domains, such as proprietary codebases.

0
ProgrammingDEV Community ·

Gemini 4 Argon, Carbon, Hydrogen and Neon Compared on Reasoning, Coding and Context

A DEV Community comparison reviews Google's four Gemini 4 models, Argon, Carbon, Hydrogen and Neon, across reasoning, scientific work, coding agents, multimodal understanding and cost per accepted result. Neon and Carbon both score 60 on the Artificial Analysis Intelligence Index at high reasoning effort, yet Neon is estimated to need about 14.8 minutes per weighted task while Carbon needs over half an hour. The article notes that all four share a roughly million-token context window, but Carbon and Argon differ notably on long-context relationship tracking, scoring 96.2% and 84.2% on a GraphWalks slice. The author argues that context capacity, the ability to follow relationships within that context, and compute cost should each be measured separately. The piece is based on published evidence as of November 19, 2026, and Modelflare did not rerun any evaluations.

0
ProgrammingDEV Community ·

Why Some Base64 Data URI Images Fail to Decode

A data URI embeds an image as text within a single string, combining a header, an encoding flag and the encoded payload. Most decoding failures come from one of three causes: a malformed header, a damaged payload, or bytes that are not actually an image. Omitting the ;base64 flag makes browsers treat the payload as plain text, so the image silently fails to appear. Missing or misplaced commas also break parsing, since the separator marks where the header ends. The declared media type is only a hint that browsers do not verify, so checking the file's leading signature bytes is the more reliable way to identify the real format.

0
ProgrammingDEV Community ·

Brute-force RDP attack on Windows Server 2019 jump host locked out an engineer after stale allowlist and alerting gaps

An engineer discovered an active account lockout on a Windows Server 2019 RDP jump host, prompting forensic review of the Security log. The logs showed roughly 40,000 failed logon attempts overnight that went unnoticed because alerting had been suppressed. A stale CIDR block in the perimeter allowlist let external scanners complete connections to port 3389, pushing authentication requests into the host's authentication subsystem. The flood consumed CPU and storage I/O and eventually exhausted the account's lockout threshold, blocking valid credentials. The incident was attributed to outdated perimeter rules combined with alert tuning that lacked rate-based burst detection.