Developer finds nine bugs that hid Python benchmark data for over a year

A developer running a multi-language Agent-to-Agent performance benchmark discovered that Python results had been missing for over a year due to CPython 3.11's default 4,300-digit limit on integer-to-string conversion, which caused crashes when computing the 24th Mersenne prime. The stringified value was never actually used in the benchmark output, meaning the fix was simply deleting the unnecessary conversion. Eight additional bugs were uncovered during the investigation, including a regex that silently failed to parse Gemini's inconsistently phrased timing responses, Node.js and Rust agents falsely reporting they had computed up to 100 Mersenne primes when only 26 exist, and a chart that grouped direct-handler agents alongside LLM-routed agents as if they shared the same pipeline. All nine issues were addressed across four pull requests, restoring accurate Python timing data and improving measurement reliability across the benchmark suite.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in