Developer finds his bot-detection classifier was confirming bias, not measuring it

A software developer on DEV Community noticed his follower count was rising sharply while post views remained unusually low, prompting him to build a classifier using the public API to identify fake accounts. His first run flagged roughly 95% of followers as bots, a result that matched his theory so cleanly it should have raised suspicion. When he tested the same classifier against 14 known-real accounts as a control group, he found that his strongest signals — empty bio, location, and website — could not distinguish bots from accounts that had simply just signed up. One indicator scored nearly identically across both groups, meaning it was measuring nothing at all yet had been read as corroborating evidence. The episode illustrated a core flaw in self-validation: a measurement that instantly agrees with your hypothesis has only been asked to confirm it, not to challenge it.