Developer finds three data leakage flaws in ML model that appeared to validate cleanly

A developer building a computer vision model to estimate container fill levels discovered three separate data leakage issues after the model appeared to perform well, with a mean absolute error of 0.055. Despite having grouped train/test split guardrails in place from the start, leakage occurred because a grouping column contained incorrect scene IDs instead of unique container IDs, meaning the split was not separating data as intended. A second leak arose when evaluation scripts bypassed the safe split module entirely and used frame-level rather than container-level grouping. A third issue involved a cross-site validation claim that broke down because a drone's default filename prefix appeared across multiple sites, causing training and test data to intermingle. The developer noted that while one published validation claim was invalidated, relative performance comparisons between model approaches remained reliable since all experiments ran on the same flawed partitions.