Vision LLMs Paired With Playwright Aim to End Brittle Web Automation

End-to-end test suites and web scraping pipelines have long suffered from fragile, hardcoded locators such as XPath expressions and CSS selectors that break whenever a frontend component changes. Modern single-page applications, randomized CSS class names, and shadow DOMs have made these traditional DOM-based selectors increasingly unreliable. A new approach called sight-driven automation combines headless browser engines like Playwright with multimodal Vision Large Language Models that interpret the browser viewport visually rather than querying the DOM directly. By capturing live screenshots and passing them to a Vision LLM, the system can locate interactive elements using visual cues — such as placeholder text — regardless of underlying HTML changes. Proponents argue this cognitive layer creates more resilient, self-healing automation that reduces the time engineering teams spend updating broken test suites.