Free-tier rate limits can silently corrupt LLM benchmark scores, developer warns

A developer building a natural language-to-SQL tool discovered that two benchmark runs 90 seconds apart on the same code produced wildly different scores — 17/20 versus 6/20 — not due to any model change but because the second run exhausted free-tier API quotas. The 14 missing SQL responses were near-instant circuit-breaker failures triggered by HTTP 429 rate-limit errors, meaning the model was never reached at all. The same issue appeared at larger scale, where a 500-question benchmark scored a misleading 0.214 because the majority of failures were quota-related fast-fails rather than reasoning errors. The author recommends spacing requests roughly 4 seconds apart, using checkpoint-and-resume strategies across multiple sessions, and keeping smoke tests and full evaluation runs on separate days to avoid shared quota drain. The core advice is to always report attempted-versus-total question counts alongside accuracy scores, and to treat instant failures as infrastructure problems rather than model regressions.