Benchmark reveals token overhead and hidden costs across 7 AI agent frameworks

A developer built an open-source benchmarking tool called agentic-arena to measure the real wire-level costs of popular AI agent frameworks under identical, controlled conditions. Six of the seven frameworks tested fell within a 1.15x token overhead band compared to a hand-rolled baseline loop, but smolagents consumed 3.9 times more prompt tokens due to a redundant system prompt that restated tool schemas already sent separately. The benchmark also found that five of seven frameworks failed to pass shared request timeouts to their HTTP clients, causing them to hang silently for up to 20 seconds on a one-second budget — a bug since fixed and added to CI. On retry behaviour, only smolagents survived three consecutive rate-limit errors, but did so by pausing for up to four minutes per item, which can quietly collapse batch throughput without surfacing in accuracy scores. The project is fully reproducible, with CI regenerating every measurement on a clean install from the public GitHub repository.