SShortSingh.
Back to feed

Postgres Creator: LLMs Achieve 0% Accuracy on Real Enterprise Database Queries

0
·82 views

Turing Award winner and Postgres creator Mike Stonebraker has claimed that large language models score 0% accuracy when tested on real-world production data warehouse queries, far below the 80–85% figures reported on popular benchmarks like Spider and Bird. Speaking on the Data Renegades podcast, Stonebraker explained that standard benchmarks use clean, simple datasets that bear little resemblance to the complex, idiosyncratic schemas found in actual enterprise systems. He identified four core reasons for the gap: enterprise data is absent from LLM training sets, real queries are far more complex, production schemas are messy and inconsistently named, and companies use domain-specific terminology that models have never encountered. To support his findings, Stonebraker released BEAVER, an anonymized benchmark based on four real data warehouses, urging AI researchers to test their models against realistic data. He drew a parallel to past tech hype cycles, warning that AI vendors may be shipping demos of capabilities that do not yet function reliably in production environments.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

How to safely unsave Reddit posts and export your saved data first

Reddit does not appear to offer an official bulk unsave or multi-select option, so the documented method is unsaving items one at a time. Because unsaving has no documented undo, the article recommends requesting an official data export first, which Reddit says can take up to 30 days. Unsaving only removes an item from your saved list and does not delete the post or comment itself. The Saved page also appears to show a limited number of items, so older saves may not be visible there. The author advises against third-party scripts or automation, which Reddit's rules restrict, and promotes their own browser extension for organizing saves.

0
ProgrammingDEV Community ·

Queue Fan-Out Outperforms Monolithic Cron for Large Email Retry Workloads

The article argues that cron should only admit work, while a queue-backed worker sends each report email with bounded retries. Each job is keyed by report date and recipient or tenant ID, so redelivery under at-least-once semantics does not cause duplicate sends. Using a hypothetical 18,400-recipient digest that hits provider rate limits partway through, the author shows how a monolithic run forces risky restarts or manual cleanup. The approach lets operators retry only unfinished recipient jobs, and alerts can focus on exhausted attempts or message age rather than transient 429 responses. The piece also notes that payloads should stay small, since queue message bodies have size limits.

0
ProgrammingDEV Community ·

Oracle Adds Assistant to Investigate Project Cost Import Exceptions

Oracle is introducing a Project Cost Exceptions Assistant for Fusion Cloud Project Costing, slated for release in 27A. Project costs that fail validation during the Import Costs process remain unprocessed, which can complicate month-end close for project accountants. The assistant lets users search, categorize, and drill into these exceptions through natural-language requests and suggests possible resolutions based on the error type. It is designed for investigation rather than correction, so corrective changes must still be made through standard Oracle Fusion functions. The distinction matters because unresolved exceptions can delay reporting and downstream billing and revenue activities.

0
ProgrammingDEV Community ·

Free side-project launch sites in 2026: sign-up hurdles, waits and badge demands

A developer listed two small web products on about 40 free launch sites and directories in November 2026, finding that writing each listing took about an hour while meeting each site's requirements took days. The sites varied widely, from simple forms and email replies to long free-queue waits of several months and requirements such as upvoting other launches or adding backlinks. Many sites also restricted free access through paid upsells, login-only sign-ups, or invite codes. The author recommends preparing a single kit file with standard names, descriptions, logos and screenshots before opening any site, and noting badge placements so they can be removed later.