Study of 300 websites finds no preferential treatment for signed AI agents
A developer built a custom tool and tested 300 public websites to verify two widely repeated claims: that sites treat cryptographically signed AI agents differently, and that some sites hide text visible only to AI models. Signed requests made no positive difference — in the three cases where signing had any effect, the signed agent was actually treated worse. Hidden machine-only text appeared on just one page out of 281, a single transparent span on firecrawl.dev directing AI agents to an onboarding file. The study also found that the median webpage costs an AI reader 2,592 tokens, while the seven sites serving markdown to machine clients cut that to around 962 tokens, though most of those markdown responses had caching header errors. A key methodological finding was that 27% of homepages returned different content between identical requests, meaning any signed-versus-unsigned comparison must account for natural page variation to avoid measuring noise.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in