Silent API fallback corrupted a developer's AI agent memory experiment

A developer testing whether memory improves AI agents found that a silent fallback had contaminated their first set of results. The experiment's model hit its daily token limit on a free tier, and failed requests were quietly replaced without being flagged. The team labeled and kept the affected data, switched to a smaller model with its own allowance, and reran the study from scratch. The clean results showed nearly identical test scores for the memory and baseline agents, with only a small difference in failed runs that the author judges too few trials to be conclusive. The author concludes that fallbacks should be logged loudly and every data point should record its provenance.