How one dev got AI search engines to actually cite his site's content

A developer at a content platform with roughly 1,600 articles investigated why almost none of their pages were being cited by AI answer engines such as ChatGPT, Perplexity, and Google AI Overviews. He found that a misconfigured robots.txt was wasting crawl budget on irrelevant endpoints, because per-bot user-agent blocks override the wildcard rule rather than merging with it. The fix involved consolidating all rules under a single wildcard directive and adding an llms.txt file — a plain-text list of the site's best URLs with brief descriptions for AI models. He also discovered that answer engines cite individual passages rather than full pages, meaning content performs better when headings are phrased as direct questions and answers are stated upfront in roughly 500-token chunks. A separate bug revealed the site's table of contents had never rendered for any reader due to a heading ID mismatch in the markdown renderer, which required re-rendering all stored article HTML to fix.