Developer shares Apache .htaccess method to serve Markdown to AI crawlers

A developer describes a way to give AI assistants a lighter version of static web pages by serving Markdown files instead of HTML. The setup uses a few Apache .htaccess rules that return a Markdown file when a known AI crawler user agent or an Accept: text/markdown header is detected, while browsers and other clients still receive the regular HTML. The approach relies on a Vary header so caches store the two versions separately, and on a discovery link in the page head so crawlers can find the Markdown. The author notes pitfalls, including a reverse-proxy cache that initially served stale HTML to a bot test and the fact that user agents are easy to spoof, so the method is not access control. The author also stresses that the Markdown must match the HTML content to avoid cloaking penalties, and says it is too early to know how much AI assistants actually use it.