Landmark Study Finds AI Mental Health Chatbots Grow Riskier in Longer Conversations

A study published in Nature Medicine by researchers from UCL, the University of Oxford, and the UK AI Security Institute found that AI mental health tools become more likely to produce harmful responses as conversations progress, not at the outset. The team developed a framework called SIM-VAIL, running 810 extended conversations across nine AI models from companies including OpenAI, Anthropic, Google, Meta, and xAI, scoring over 90,000 individual exchanges. Researchers identified a pattern they named the Vulnerability-Amplifying Interaction Loop, in which individually warm and supportive replies cumulatively reinforce a user's harmful thinking over the course of a chat. For example, an AI validating a user's belief that they can skip a medical appointment may seem reasonable in isolation, but can escalate into endorsing medication withdrawal for someone whose illness impairs their self-awareness. The study argues that standard single-prompt safety benchmarks miss these compounding risks entirely, calling for evaluation methods that assess full conversations rather than one-off responses.