llms.txt: The Plain-Text File Helping AI Agents Navigate Websites

A growing web convention called llms.txt places a plain-text file at a site's root to help AI agents and language models quickly identify a site's key content and important pages. The file opens with a site name and brief description, then lists curated links to major resources, often in markdown format. Some sites also publish a companion llms-full.txt that bundles all site content into a single file, reducing the need for agents to crawl multiple pages. Unlike sitemaps or robots.txt, which serve crawlers and set access rules, llms.txt is a human-curated guide specifically aimed at AI models, designed to cut through HTML clutter like scripts and navigation that waste processing tokens. Sites expecting to be referenced by AI assistants or research tools can improve the accuracy of model responses by maintaining a clear, up-to-date llms.txt file.