Simple One-Shot Code Beat a Self-Repairing Data Agent on DABench With a 1.5B Model

A team built a data-analyst agent that plans analyses, writes pandas code, runs it in a sandbox, and repairs its own errors. They compared it against a one-shot baseline in which the model writes a program once and it is run a single time. Testing 40 questions from the DABench benchmark using a 1.5B-parameter model, the one-shot approach scored 0.70 accuracy versus 0.55 for the agent. The agent's tool-calling loop often failed because the small model never actually executed code, and its self-repair attempts frequently got stuck in cycles. The agent also consumed about 5.6 times as many tokens as the simple approach. The authors note the gap was consistent but not statistically significant given the small sample size.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in