Developer Finds 20x Statistical Error in AI Ad-Testing Agent That Never Triggered an Alert

·1 views

A developer auditing their AI-powered ad-testing agent, Mia, discovered it had been sizing A/B tests at roughly one-twentieth of the required sample size due to misapplied statistical shorthand. The textbook formula prescribed 1,111 users per test arm, while the correct calculation for a 1.9% click-through rate demanded 22,278. Because statistical underpowering produces no runtime errors or failed assertions, the system continued returning confident winners throughout — some of which were likely false. A separate issue compounded the problem: three-arm tests were evaluated at a raw p-value threshold of 0.05, inflating the true false-positive rate to around 14%, meaning roughly one in seven declared winners did not actually exist. The developer has since updated Mia to use the exact unpooled formula at the observed base rate and applied Holm-Bonferroni correction for multiple comparisons.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

TrendVidStream Replaced SQLite Job Queues With NATS JetStream for Regional Video Events

TrendVidStream, which runs video discovery crawlers across 8 regions, replaced an inline cron-based processing system with NATS JetStream to handle events like indexing, cache invalidation, and sitemap updates. The original approach fused data fetching and fan-out into a single process, causing partial failures when any step crashed or stalled. A SQLite jobs-table queue was also evaluated but rejected due to polling inefficiency, locking contention, lack of native fan-out, and inability to work across machines without added replication. NATS JetStream was chosen for its push-based delivery, multi-consumer support, independent storage, and cluster capability — all in a single Go binary requiring no container infrastructure. The team configured a stream with subject-based regional filtering, a 72-hour message retention window, and a 2-minute deduplication window to ensure resilience during deployments.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Self-Taught Developer Reviews Cyfrin Updraft as Top Free Blockchain Learning Platform

A developer documenting a self-taught blockchain engineering journey shared a Week 2 progress update, having completed 45% of the Blockchain Basics course on Cyfrin Updraft. The platform stood out for its use of Tenderly virtual testnets, which provide instant private blockchains with unlimited test ETH, eliminating the common frustration of hunting for working faucets. The reviewer compared Cyfrin Updraft favorably against LearnWeb3.io, noting the former excels in depth of instruction while the latter offers better gamification and community features. A key conceptual takeaway from the week was understanding the oracle problem — the limitation that smart contracts cannot access real-world data on their own, which is why decentralized oracle networks like Chainlink exist. The reviewer recommended using both platforms together rather than choosing one over the other.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How a Defence Contractor Deployed AI Code Review in an Air-Gapped Environment

A defence contractor with a classified, air-gapped codebase needed AI-assisted code review without any external connectivity, ruling out all cloud-based solutions. The development team of twelve engineers was producing over 200,000 lines of code annually and struggling to maintain review quality. The solution involved deploying large language models entirely on-premise, running on locally controlled GPU hardware such as NVIDIA A100 servers. Key considerations included model size selection, 4-bit quantisation, inference server configuration, and CI/CD pipeline integration to meet strict compliance requirements. Hardware costs ranged from roughly $30,000–$45,000 for smaller teams, amortising to a figure comparable to standard cloud-based code review licensing.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Understanding C Pointers: What They Are and Why They Matter for Beginners

Pointers are one of the most challenging concepts for beginners learning the C programming language, yet the core idea is straightforward: a pointer is a variable that stores a memory address rather than a direct value. Using the & operator retrieves a variable's address, while the * operator allows access to the value stored at that address, a process called dereferencing. Pointers become essential in several practical scenarios, including passing large data structures to functions efficiently, modifying variables from within a function, and managing dynamic memory allocation at runtime. Common beginner mistakes include using uninitialized pointers, which can cause crashes or memory corruption, and dereferencing NULL pointers. Developing safe habits early, such as always initializing pointers, helps avoid difficult-to-trace bugs in C programs.

0 comments Read more at DEV Community