Study Finds Vision-Language AI Still Struggles With Complex Social Scenes Despite Benchmark Gains

A new research study evaluated nine vision-language AI models spanning 2017 to 2025 on their ability to describe complex social behavior in images, challenging the widely cited claim of near-human captioning performance. Researchers created a 100-image Complex Social Behavior dataset drawn from movie frames requiring multi-person interaction reasoning, alongside a ranked set of 20 human descriptions per image as a gold standard. Results showed that older pre-MLLM models performed poorly on the new dataset despite appearing competent on the standard MS-COCO benchmark, while modern multimodal large language models closed much of that gap. The study introduced a five-category error taxonomy and found that modern models have largely eliminated four error types, but continue to struggle with spatial dependence — understanding positional relationships between objects and people. The findings suggest that headline accuracy figures in AI vision research are significantly inflated by easy benchmark datasets that fail to capture real-world scene complexity.