Randomized Experiment Failed Not Due to Coin Bias But Missing Data Infrastructure

A team ran a cryptographically fair randomized experiment to establish causal claims about a system change, assigning tasks to two arms via a provably unbiased coin flip. Weeks into the experiment, an audit revealed the results could never be analyzed — not because the randomizer was flawed, but because the assignment log never recorded which task each coin flip was tied to. Without a join key linking each draw to the unit assigned and its outcome, the experiment's data existed in two unconnected piles with no way to pair them. The authors identify three non-randomizer requirements for a valid experiment: a join key recorded at the moment of assignment, a safeguard against selective re-drawing, and a pre-experiment power calculation. The core lesson is that verifying the coin covers only a fraction of what makes a randomized experiment work — the rest is unglamorous but critical data bookkeeping.