Developer's 22 Passing Tests Still Missed a Real-World API Bug — Here's Why
A developer with 22 passing tests and two code reviewers discovered a critical bug only when integrating with a live third-party API, where the parser expected ISO 8601 timestamps but the API returned Unix seconds. The flaw went undetected because both the parser and its test fixtures were written by the same person, encoding an identical wrong assumption, making the suite internally consistent but disconnected from reality. A separate security issue was also uncovered: Python 3.13's default redirect handler preserved API keys in headers across origins, a problem solvable with Request.add_unredirected_header(), which the developer had not used. The bugs were caught only through a real integration test conducted in collaboration with FIPSign, a third-party API provider who supplied disposable keys and stayed engaged for ten weeks. The episode highlights that internal test consistency does not guarantee correctness, and that live external validation can surface assumptions no amount of internal review will catch.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in