Postgres Creator: LLMs Achieve 0% Accuracy on Real Enterprise Database Queries

Turing Award winner and Postgres creator Mike Stonebraker has claimed that large language models score 0% accuracy when tested on real-world production data warehouse queries, far below the 80–85% figures reported on popular benchmarks like Spider and Bird. Speaking on the Data Renegades podcast, Stonebraker explained that standard benchmarks use clean, simple datasets that bear little resemblance to the complex, idiosyncratic schemas found in actual enterprise systems. He identified four core reasons for the gap: enterprise data is absent from LLM training sets, real queries are far more complex, production schemas are messy and inconsistently named, and companies use domain-specific terminology that models have never encountered. To support his findings, Stonebraker released BEAVER, an anonymized benchmark based on four real data warehouses, urging AI researchers to test their models against realistic data. He drew a parallel to past tech hype cycles, warning that AI vendors may be shipping demos of capabilities that do not yet function reliably in production environments.