A/B Testing Guide for Data Analysts: Statistics, Pitfalls, and Best Practices

·1 views

A/B testing is a widely used experimentation method where users are split into two groups to compare the performance of two versions of a product or feature. While the core statistics are relatively straightforward, analysts must apply careful discipline around hypothesis setting, sample size calculation, and result interpretation to avoid costly errors. Common mistakes include underpowered tests, peeking at results early, and confusing statistical significance with practical significance. Reading confidence intervals alongside p-values is recommended for a more complete and accurate picture of any measured effect. A/B testing is also a frequent topic in data analytics interviews, particularly for roles in product, e-commerce, and marketing analytics.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Ghost database from cancelled CI job silently faked a passing migration test

A database migration at an unnamed engineering team appeared to pass all pipeline checks in July but failed immediately in staging because it relied on a Postgres extension that was never explicitly installed. The root cause was traced to long-lived CI runner virtual machines where Docker Compose reused containers from previous jobs, including one from a different branch that had been cancelled mid-run three days earlier. Because the teardown step only executed on success, the cancelled job left its database — complete with the installed extension — running on the shared host. An audit of the four machines uncovered 61 orphaned containers and 140 volumes, some dating back to February. The team resolved the issue by switching to ephemeral runners destroyed after each job, scoping Compose project names to the job ID, running teardown unconditionally, and adding a preflight check to ensure no pre-existing containers are present before a job starts.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Company finds 19 active shared mailboxes unmonitored, holding 1,400 unread emails

A customer complaint in April revealed that a shared mailbox printed on delivery notes had gone unmonitored for over a year after a mail migration broke its forwarding rule. An internal audit of 212 shared mailboxes found 64 unopened in 90 days, with 19 still receiving external emails and collectively holding around 1,400 unread messages, including carrier claims, customer cancellations, and council enquiries. The root cause was traced to a mailbox creation process that required no named owner or responsible manager. In response, the company overhauled its policy: every shared mailbox must now have an owning team and named manager, owners must confirm their mailboxes twice a year, and externally facing mailboxes are monitored so any message unread for five working days triggers a ticket. The original complainant, who had waited seven weeks for a response about a damaged pallet, was ultimately refunded.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Exposed .git Folders in Public S3 Buckets Can Leak Entire Source Code Histories

A vulnerability disclosed via HackerOne (report #2383486) showed that Mozilla accidentally uploaded its .git/ directory alongside marketing content to a public AWS S3 bucket. Because the deployment pipeline used git clone rather than a clean export, the entire repository metadata — including commit history, developer emails, and git remote URLs — was publicly accessible. Attackers can reconstruct the full repository, including secrets that were previously deleted from later commits, using just three shell commands. The root cause is a mismatch between developer intent and pipeline behavior: aws s3 sync uploads everything in the build directory, not just the intended build output. The fix is straightforward — either delete the .git/ folder before syncing or use git archive, which exports only tracked file content without repository metadata.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Finds 532 Stale URLs Two Weeks After GitHub Repo Transfer

A developer transferred an open-source mental health toolkit from an old GitHub account to a new named account, assuming the platform's automatic redirects would handle everything correctly. However, a routine SEO audit two weeks later revealed 532 hardcoded URLs across 47 HTML files still pointing to the old domain. The stale references affected Open Graph image tags, canonical URLs, and JSON-LD structured data, causing duplicate content signals for search engines and occasional broken preview cards on social media. A short Python script resolved all 532 instances in a single commit by replacing the old domain with the new one. The developer has since added a CI check to catch any recurrence and recommends using relative URLs and a centralized base URL config to avoid similar issues after future transfers.

0 comments Read more at DEV Community