Flawed Ablation Design Left Engineer Unable to Isolate What Code Change Actually Shipped

A software engineer discovered that a four-arm ablation experiment failed to isolate the intended code change because a single boolean flag silently encoded two independent decisions at once. The flaw was not in the replay mechanism or the archive, which were verified and reproducible, but in the ablation design itself. The fix under study involved a Stop hook in a coding harness that checks whether an AI agent folds under user pushback, but the hook's backward search could be misled by harness-injected entries masking genuine human challenges. Two separate fixes existed for this blind spot, yet no single ablation arm toggled exactly one of them in isolation. The post argues that even a technically sound replay setup cannot compensate for a confounded experimental design, where the comparison between arms does not cleanly reflect a single variable.