Developer finds nine bugs that hid Python benchmark data for over a year

A developer maintaining a multi-language Agent-to-Agent benchmark discovered that Python results had been missing for over a year due to CPython 3.11's 4,300-digit limit on integer-to-string conversion, which caused crashes when computing large Mersenne primes. The root bug was a redundant str() call on a prime number exceeding 6,000 digits, even though the stringified value was never actually used in the output. Eight additional issues were uncovered during the fix, including a regex that silently failed to parse LLM-generated timing prose, agents falsely reporting they computed more Mersenne primes than the known table allows, and a benchmark chart that mixed two fundamentally different pipeline architectures without distinction. Fixes were submitted across four pull requests, correcting timing methods, output parsing, result reporting accuracy, and chart labeling. The developer noted the core lesson: benchmark data should never rely on how a language model chooses to phrase a sentence.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in