Study finds BIRD benchmark marks text-to-SQL models wrong when their answers are correct

An audit of the BIRD-dev text-to-SQL benchmark revealed that 19% of model mismatches stemmed from the model producing more accurate SQL than the benchmark's own reference answers, particularly around the use of DISTINCT in COUNT queries. Researchers found that gold-standard annotations sometimes used COUNT without DISTINCT on one-to-many joins, causing over-counting — while the model's corrected version was penalized by the automated scorer. Independent research from the Kang lab at UIUC, published ahead of VLDB 2026, found that 52.8% of BIRD benchmark instances contain annotation errors including wrong SQL, ambiguous questions, and schema mismatches. The authors warn that blindly optimizing against flawed benchmarks can lead developers to remove correct SQL patterns, degrading real-world query accuracy in pursuit of higher leaderboard scores. They recommend bucketing and auditing all model losses before writing any prompt directives, to distinguish cases where the model is genuinely wrong from cases where the benchmark's answer key is at fault.