Few AI crawlers fetched llms.txt in early server logs, favoring Markdown page copies
A new website's server logs over about 75 hours show that only three verified crawlers, Googlebot, GoogleOther and ClaudeBot, requested llms.txt or llms-full.txt, with six such requests in total. Those same crawlers fetched the site's Markdown page copies 77 times, which is far more than the llms files. Bingbot, Applebot, and the OpenAI and Perplexity crawlers never requested the llms files during the window. The crawlers that did read llms.txt had already fetched Markdown copies, suggesting they did not discover those pages through it. The analysis counted only requests from IP ranges published by each crawler's company, and the llms.txt proposal itself is not a formal standard adopted by search engines or AI firms.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in