Audit of llms.txt and robots.txt Checks Policy Evidence Without Predicting AI Citations
A website's llms.txt file does not determine whether it appears in a given AI answer, and a missing file does not mean public content is inaccessible. The llms.txt Auditor build 0.3.2 examines llms.txt, llms-full.txt, and root robots.txt for submitted origins, reporting observed policy evidence and gaps rather than indexing or citation outcomes. The tool distinguishes definitive missing-file responses from timeouts, access denials, rate limits, and server errors, which are treated as incomplete evidence. Root crawler rules are reported as blocked, allowed, or unspecified for a curated set of AI and search bots, with unspecified not treated as permission. The article stresses that the audit cannot measure crawler visits, brand mentions, citation frequency, or traffic, which require separate observations.
