Author benchmarks own agent dataset against four others and reports losing on several metrics

A developer compared their 66-session agent trajectory dataset, MegaBit, against four larger datasets using a uniform evaluation. They reported losing on size, repository count, retry-chain length, and on a recovery-density metric they devised themselves, where SWE-Gym scored higher. The author states they won on tool calls per row, assistant turns, audit checks, and the share of rows containing a real failing run followed by a retry. The dataset is MIT-licensed, comes with an offline verifier, and the author invites others to test the metric and identify flaws in it.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.


Discussion (0)
Log in to join the discussion and vote.
Log in