AI crawlers ignored llms.txt but actively fetched Markdown page copies, logs show

A developer added both an llms.txt index file and per-page Markdown copies to their website to improve AI discoverability, then logged all requests to both files over eight days. The Markdown twins received 30 fetches from legitimate AI crawlers, led by Meta's bot with 24 requests, followed by Amazon, Anthropic, and OpenAI bots. The llms.txt file, by contrast, received no requests from any AI crawler — only two automated scanners and two likely human browser visits. The author notes a key structural difference: Markdown twins serve visitors who already found the page, while llms.txt requires crawlers to proactively seek it out. Google's own developer docs reflect the same pattern — offering Markdown versions of pages while publishing no llms.txt of their own.