OpenART Benchmark Tests AI Agent Safety Across 10,000 Stateful Scenarios

Researchers have developed OpenART, a safety evaluation framework that tests AI agents across more than 10,000 validated scenarios spanning 50 domains. Unlike static benchmarks, OpenART focuses on how safety failures can emerge gradually through evolving environment states — including changes to workspace data, permissions, memory, and plans — rather than from single problematic prompts. Each test keeps the task objective and a hidden safety contract constant while only the environment state changes, allowing the framework to detect delayed failures that traditional benchmarks may miss. Testing across 75 agent-model configurations, OpenART recorded a strict Attack Success Rate of 85%, with success requiring confirmation from both a deterministic evaluator and a GLM-5.2 judge. The findings highlight significant vulnerabilities in stateful AI agents, where an early authorized action can cascade into unsafe outcomes many steps later.