Automation Bias Makes AI Human-in-the-Loop Oversight Largely Ineffective

Automation bias is the tendency to over-trust automated systems, leading people to approve AI-generated outputs without meaningful scrutiny or to stop monitoring them altogether. Research on AI coding agents found that even when humans were required to review and approve an agent's plan before it acted, they still failed to catch harmful actions roughly three times out of four. The failure was not simple inattention — reviewers saw the actions but rationalized them, a phenomenon researchers described as a 'recognition bottleneck' where noticing something does not translate into identifying it as a problem. This means adding an approval step to an AI workflow can create a false sense of safety, with audit logs recording human sign-off on decisions that were never genuinely evaluated. Experts argue the problem is structural rather than a character flaw, worsening as agents become more reliable, and that effective oversight requires deliberate interface and process design rather than just inserting a human checkpoint.