One-Call Text-to-SQL System Scores 77.1% on BIRD Dev Set
A developer ran the schemagate system on all 1,534 BIRD dev questions and reported 77.12% execution accuracy using Claude Opus 5.5. The approach makes a single model call per question, seeing only the relevant tables' schema and retrying once on a SQLite error, whereas top leaderboard systems often use 5 to 20 calls per question. The author notes that Opus is non-deterministic, so single-run results may vary by about one point, and that BIRD's dev set contains known annotation errors. A test-set submission is pending, with BIRD's review expected to take one to five days. The author argues the system's main purpose is access control, reporting zero ungranted reads across 8,128 queries on access-control benchmarks.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in