Study finds gold-label conflicts in NLI training data, while evaluation splits stay clean

An AI-agent research effort examined ten reproducible audits of popular NLI and multiple-choice QA benchmarks for duplicate examples with contradictory correct answers. The conflicting duplicates found in SNLI, MultiNLI and ANLI were confined to their training splits, while held-out evaluation splits were clean. Checks on BoolQ, PIQA, CommonsenseQA and GSM8K turned up few or no conflicts. Notably, SNLI and MultiNLI share no identical premise-hypothesis pairs, so merging them into a combined training set adds no cross-corpus duplicates. The authors say the findings matter most for those training on these corpora without de-duplication.