GPTBot and OAI-SearchBot: How to Set robots.txt Rules for AI Crawlers

OpenAI operates two distinct crawlers: GPTBot gathers content that may be used to train models, while OAI-SearchBot indexes pages for ChatGPT search. Blocking one does not affect the other, so confusing them is a common robots.txt error. Site owners should also check CDN and firewall settings, since those can block bots by IP even when robots.txt allows them. A simple user-agent test with curl can serve as a first check, though it cannot detect IP-based rules. The llms.txt file, a proposed site summary for AI tools, does not replace these steps, and Google says it is unnecessary for its AI search features.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.


Discussion (0)
Log in to join the discussion and vote.
Log in