Developer Runs 48-Hour AI Prompt Regression Test, Finds Output Consistency Unreliable

·1 views

A developer discovered that a free AI model returned three different action plans for the same prompt within one hour, each delivered with equal confidence, prompting a structured reliability test. To assess consistency, they froze a battery of 11 prompts and ran each three times every six hours over 48 hours, recording all raw outputs without adjusting anything mid-run. A diversity ratio metric was used to score output stability, where a perfect repeat scored 0.33 and fully inconsistent outputs scored 1.0. Contrary to expectations, short prompts initially appeared stable but then abruptly changed key names, while long prompts showed high variability from the very first run. The most problematic finding was not semantic drift but "serialization drift" — the same JSON prompt returning raw JSON, fenced code blocks, or preamble-wrapped text across different runs, breaking parsers silently.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How Well-Written Pull Requests Make Code Review Faster and More Effective

A poorly documented pull request with a blank description, a vague title, and 600 lines of unexplained changes once slipped past review and crashed a production checkout flow. The incident highlights how PRs are often treated as bureaucratic checkboxes rather than collaborative conversations, undermining their core purpose. A pull request is a formal request to merge code changes into a main branch, but its real value lies in the social layer — the discussion, decisions, and context it preserves for future developers. Experts recommend that PR titles clearly state the change in imperative form, while descriptions should answer why the change was made, what it includes, and how to test it. Following these practices can significantly reduce review turnaround time and improve overall code quality across engineering teams.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why Smart Contract Token Approvals Are Transactions, Not Toggles

Token approvals on ERC-20 contracts are stored on-chain as allowance entries, not app-level settings, meaning any system relying on them must treat that on-chain record as the sole source of truth. Revoking an approval is itself a full blockchain transaction that must be signed, broadcast, and confirmed — it is not instant and can be delayed by network congestion. Services that cache approval states locally risk acting on stale data, either attempting to spend tokens already revoked or blocking valid spends when a revocation is still unconfirmed. The safest engineering practice is to read the allowance directly from the chain immediately before executing any token transfer, rather than relying on a database record. This approach adds RPC latency and infrastructure load but prevents a class of bugs where a system's internal state diverges from what the token contract will actually permit.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Attackers Can Silently Disable AWS CloudTrail Logging With a Single API Call

Security researchers warn that an attacker who gains access to an AWS account can halt audit logging with a single command — aws cloudtrail stop-logging — leaving the trail's configuration visually intact while new events stop being recorded. The technique, catalogued under MITRE ATT&CK as T1562.008, is well-documented in Mandiant incident reports and AWS security guidance. Because standard inspection commands like describe-trails and GetTrail do not reveal the logging state, only get-trail-status exposes the critical IsLogging: false flag. Inside this blind window, attackers can exfiltrate secrets, create persistent backdoor IAM users, and rotate KMS keys to prevent decryption of existing log archives — all without generating a CloudTrail record. Security teams are advised not to treat 'trail configured' as equivalent to 'trail running,' and to actively monitor the logging status field rather than relying solely on static configuration audits.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Silent calendar bug corrupted five years of Korean astrology charts, tests missed it

A developer building a Korean saju (BaZi) astrology service discovered that a third-party calendar library was returning the same solar term timestamp for every year, regardless of the year passed as input. The bug meant that birth charts for anyone born on or near a solar term boundary between 2020 and 2030 — except 2026 — may have been calculated incorrectly, sometimes by as much as twelve hours. The flaw went undetected for five years because the test suite's expected values had been derived from the same faulty library, making the tests confirm the library's output rather than astronomical reality. A secondary issue compounded the problem: all test probes had been chosen on the same side of the real boundary, so they would have passed even against an independent correct dataset. The developer has since replaced the dependency with verified astronomical data and redesigned the tests to assert that different years must produce distinct solar term instants.

0 comments Read more at DEV Community