Developer Builds Simple Failure Classifier After 48 Hours of Misdiagnosing AI Errors

A developer running a 48-hour experiment with a free AI model endpoint discovered that misclassifying failures was a bigger problem than the failures themselves. Over two days, they logged 21 failure events — defined as final attempts that failed after three retries — from a Python worker making timed calls to a free model endpoint. Manual notes proved unreliable, with the same symptom labeled differently within a single day, prompting the developer to build a deterministic script to sort failures into three categories: code issues, model or API problems, and server-side transients. The classifier revealed that nearly half the events had been mislabeled in the original notes, with seven being harmless server blips already handled by the retry loop. The key takeaway is that establishing fault before attempting a fix can prevent wasted effort on code that was never broken.