SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

How to Set a Minimum Detectable Effect That Actually Works for A/B Tests

0
·2 views

When setting up an A/B test, the Minimum Detectable Effect (MDE) is the smallest true improvement a test is designed to reliably identify, and it directly determines how much traffic and time an experiment requires. Unlike other inputs in a sample-size calculator, MDE has no default correct value, making it a business judgement as much as a statistical one. A common source of confusion is that most calculators express MDE as a relative lift rather than an absolute percentage-point change, meaning a 10% MDE on a 20% baseline conversion rate targets a 2 percentage-point shift, not 10 points. Setting the MDE too small can leave a test running indefinitely, while setting it too large risks missing meaningful improvements the business actually cares about. Understanding the relationship between MDE, baseline conversion rate, and sample size is essential to designing experiments that are both feasible and informative.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Developer builds Cosmos, a productivity-focused new-tab browser workspace

A developer has created Cosmos, a custom new-tab page extension designed to replace the default blank browser tab. The tool targets developers and designers by offering a workspace with app shortcuts, bookmarks, and useful information in one place. Cosmos features purpose-built widgets tailored to developer and designer workflows, a combination the creator says has not been well executed before. The project is hosted and accessible via Vercel, suggesting it is currently in a public or demo stage. The announcement was shared on DEV Community, positioning Cosmos as a productivity tool for technical and creative users.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Multi-Armed Bandit Testing: Smarter Traffic Allocation Than Standard A/B Tests

A multi-armed bandit is an algorithm that dynamically reallocates website traffic toward better-performing variations in real time, rather than maintaining a fixed split until a test concludes. Unlike traditional A/B testing, which prioritizes statistical precision before declaring a winner, bandit algorithms balance exploration of untested options with exploitation of the current best performer. Three core algorithms drive most bandit implementations: epsilon-greedy, Thompson sampling, and Upper Confidence Bound, each offering a different strategy for managing the explore-exploit tradeoff. The key metric guiding these algorithms is 'regret' — the gap between conversions actually earned and the maximum possible had the best variation been served throughout. Bandit testing is best suited for scenarios where maximizing conversions during the experiment matters more than obtaining a clean, unbiased statistical result.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

JetBrains Rider Bug Silently Fails to Save Unity Assembly Definition References

A developer working on a Unity tactical game discovered that JetBrains Rider's Alt+Enter shortcut for adding assembly definition references appeared to work but left .asmdef files unchanged. The issue surfaced after integrating UnityAtoms via the Package Manager, where types like AtomEvent remained unresolved despite seemingly successful imports. Standard troubleshooting steps — including regenerating .csproj/.sln files and clearing the Library folder — failed to resolve the problem. Manually editing the .asmdef file directly in Rider offered only an inconsistent workaround, with references sometimes reverting without a clear trigger. The bug has since been linked to a known JetBrains issue tracker report, RIDER-94834, confirming the problem affects other users as well.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

How to Pick an AI Text Classification API for E-Commerce Hiring Pipelines

A developer building an e-commerce backend for scoring job candidates recommends choosing an AI classification API based on JSON contract reliability, per-tenant cost visibility, and model-switching flexibility rather than general benchmarks. The core metric proposed is schema-conforming classifications divided by attempted classifications, treating any output that fails JSON Schema validation as a failed call. OpenAI, Claude, and Gemini should all be evaluated on the same held-out set of real, previously reviewed candidate records spanning sparse, lengthy, and multilingual résumés. Per-tenant usage tracking is emphasized as critical, since large tenants can skew aggregate cost data and obscure billing accountability. The author also presents a buy-versus-build comparison of direct provider integrations versus a portable routing layer, advising teams to set accuracy thresholds from their own error budgets rather than universal standards.

0 comments Read more at DEV Community