llms.txt: The Markdown File That Helps AI Crawlers Navigate Your Website

An llms.txt file is a plain Markdown document hosted at a website's root domain that gives AI crawlers a structured, curated overview of a site's most important pages. The concept was proposed by Jeremy Howard in 2024, based on the idea that language models perform better with clean, contextual summaries than with raw HTML. A standard llms.txt file includes a site name, a brief blockquote summary, optional notes, and categorised links with short descriptions. Unlike robots.txt or a sitemap, it is written specifically for machine readers rather than search engine indexing or human navigation. No major AI provider has confirmed that the file directly influences rankings or citations, but proponents argue it is a low-cost, low-risk practice that reduces ambiguity for AI models parsing web content.