How One Recruiter's Bias Quietly Made an AI Hiring System Less Effective

A company's AI interview platform showed consistently improving metrics over six months, with pass rates, completion rates, and candidate satisfaction all trending upward. However, a senior engineering manager noticed that despite more candidates clearing interviews, the actual on-the-job performance of new hires showed no improvement over previous cohorts. Investigations ruled out AI tool usage by candidates, question leakage, and model drift as causes. The team eventually traced the issue to a feedback loop in the system's design, where human reviewer approvals were used monthly to recalibrate AI scoring thresholds. One recruiter who consistently approved nearly 90% of AI recommendations — far above the team average of 55–65% — had inadvertently skewed the model toward her preferences at scale, gradually lowering the effective bar for all candidates.