Study finds all major text-to-SQL benchmarks ignore role-based access control

A new research paper titled 'Benchmarking Text-to-SQL under Role-Based Access Control' by Yang Fei and colleagues reveals that widely used benchmarks like Spider, BIRD, and LiveSQLBench evaluate AI systems without any user permission constraints. The researchers built a dataset spanning 53 databases, 399 tables, and over 21,500 role-annotated query instances with access policies defined at the individual column level. Testing existing text-to-SQL systems against this dataset showed significant performance degradation, with many generating SQL queries that are technically correct but violate access control rules — a category the paper terms 'RBAC-rejected' queries. The core problem identified is that standard AI pipelines apply database access controls only at query execution, meaning restricted tables and columns are still visible to the model during SQL generation. The paper argues that access control must be enforced earlier, at the schema-selection stage, so that restricted objects are never presented to the model at all.