SynthDocBench Benchmark Exposes Key Weaknesses in Vision Language Models

Researchers have introduced SynthDocBench, a fully synthetic benchmark designed to evaluate how Vision Language Models (VLMs) handle long, complex visual documents. Unlike existing benchmarks such as DocVQA and ChartQA, it isolates variables like document length, layout structure, and question type to pinpoint specific model failures. The benchmark uses an LLM pipeline to generate documents across six layout archetypes, with a 40 percent random override to prevent models from exploiting predictable patterns. Testing on seven frontier VLMs revealed that performance drops sharply as document length increases, and that models struggle most with content located in the middle third of a document. A consistent early-to-late performance decline was also observed, highlighting significant gaps in how current VLMs process extended visual content.