Near-duplicate images can inflate machine learning test scores, study and tool show

A test score can overstate a model's real performance if test images are near-copies of training images, since the model has effectively already seen them. A roughly 50-line Python script can detect most such copies by shrinking each image to a small grey thumbnail and comparing thumbnails by similarity. Researchers found that 3.3% of CIFAR-10 test images and 10% of CIFAR-100 test images have duplicates in the training set. Replacing those images with new ones lowered accuracy of popular models by 9 to 14 percent relative to the original results. Common sources of such overlap include repeated shots of the same subject, frames from one video, resaved or resized copies, and merged folders, which random file-name splits cannot catch. The authors recommend reviewing every flagged pair manually and splitting data by part or photo session rather than by individual file.