Set-of-Mark prompting improves AI agent accuracy for UI interaction

Frontier vision models often struggle to accurately pinpoint clickable elements on a screen by pixel coordinates, causing failures in automated workflows. The Set-of-Mark technique, introduced by Microsoft Research, overlays numbered labels on interface elements, turning coordinate regression into a more reliable multiple-choice task. This method leverages the Document Object Model in web browsers to automatically label interactive items like buttons and inputs. Scaling errors from image preprocessing are a common source of inaccuracy, separate from model capability. The approach ensures AI agents click the correct target by matching a predicted number to a pre-mapped screen location.