Checking an ML benchmark for train/test duplicate rows with a hash-based audit

A small time series dataset, PowerCons, has 180 training and 180 test rows, and 67 of the test rows appear in the training set unchanged. A 1-nearest-neighbour classifier scores perfectly on those duplicates, which inflates results without any visible error. The author proposes an efficient check: fingerprint each row by hashing its contents with the label excluded, then look up test fingerprints in a set built from the training rows. A small Python script that reads the archive's .ts files runs this check in a fraction of a second. Applied to the UCR and UEA archives, it finds leakage in 17 of 128 UCR datasets and 3 of 30 UEA datasets on their default splits.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in