robots.txt, sitemap.xml, and llms.txt: What Each File Actually Does

Three small text files serve distinct but complementary roles in how bots and AI systems interact with a website. The robots.txt file, standardized as RFC 9309 in 2022, acts as an access policy telling well-behaved crawlers which paths they may or may not fetch, though it cannot force deindexing of already-known pages. The sitemap.xml file lists URLs a site owner wants search crawlers to consider, along with optional metadata like last-modified dates, but listing a URL is a suggestion rather than a guarantee of indexing. Introduced in 2024, llms.txt is a markdown file that gives AI assistants a curated, structured overview of a site's most important content, helping them avoid crawling an entire site to answer user queries. Unlike robots.txt, which governs access, llms.txt is purely a content-curation guide and does not restrict or grant any crawling permissions.