Postgres Creator: LLMs Achieve 0% Accuracy on Real Enterprise Database Queries

Turing Award winner and Postgres creator Mike Stonebraker has found that large language models score 0% accuracy when tested on real-world enterprise data warehouse queries, far below the 80–85% figures reported on popular benchmarks like Spider and Bird. Stonebraker tested leading LLM-based text-to-SQL systems against four actual production data warehouses, publishing the results as the BEAVER benchmark on arXiv. He attributes the failure to four core issues: enterprise data is absent from LLM training sets, real queries are far more complex than benchmark queries, production schemas use cryptic and inconsistent naming conventions, and enterprises rely on domain-specific terminology that models have never encountered. Even with techniques like retrieval-augmented generation and providing explicit schema hints, accuracy only reached a maximum of 35%, compared to over 90% for a skilled human SQL programmer with schema access. Stonebraker urged AI researchers to test against realistic benchmarks, warning that current text-to-SQL products are being shipped as capable despite not functioning in production environments.