New Controlled Benchmark Pinpoints Why Language-Model Data Attribution Fails

Researchers examined why methods for attributing language-model outputs to training data often break down, pointing out that existing evaluations tend to change several dataset properties at the same time. To address this, they introduced CATT, a controlled fact-tracing benchmark that fine-tunes models on wholly fictitious knowledge, so the correct answers are unambiguous. Such attribution is considered important for tracing outputs back to their sources, auditing data, and diagnosing model failures. The authors suggest the field validate methods on controlled synthetic benchmarks before claiming progress on real, confounded training data.