Over half of text-to-SQL benchmark answers are wrong, study finds

A UIUC research team audited leading text-to-SQL benchmarks and found that 52.8% of BIRD Mini-Dev annotations and 62.8% of Spider 2.0-Snow annotations contained incorrect answer keys, as confirmed by human SQL experts. The errors stemmed from annotators misreading unfamiliar database schemas, applying wrong domain knowledge, and misinterpreting ambiguous questions. When just 100 examples were corrected and 16 open-source agents re-evaluated, relative performance scores shifted by up to 31% and leaderboard rankings moved by as many as 9 places. In response, one developer proposed inverting the benchmark-building process by declaring ground-truth answers as constraints first and then generating a database guaranteed to satisfy them. This approach eliminates annotation error by ensuring expected query results are mathematically built into the data rather than derived by humans reading existing rows.