How to check and set robots.txt rules for AI search and training crawlers
A DEV Community guide from RankControl explains how to inspect a site's robots.txt for rules that may block AI crawlers, using a simple curl and grep command. It notes that AI companies run separate crawlers for search answers, model training, and single-page retrieval, each needing its own rule. The article recommends allowing search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot while disallowing training crawlers like GPTBot, ClaudeBot, Google-Extended, CCBot and Bytespider. It also warns of common errors, including a wildcard Disallow under User-agent: * and copying outdated GPTBot blocks, which can silently affect AI visibility.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in