Cloudflare Defaults May Block ChatGPT Training Crawlers on Your Site
A developer audit of 286 portfolio sites found that 19 blocked GPTBot and ClaudeBot, and 17 of those blocks were added by Cloudflare's dashboard rather than by site owners. The article explains that AI crawlers serve different purposes: training crawlers like GPTBot and ClaudeBot collect content for future models, while search and retrieval crawlers like OAI-SearchBot and PerplexityBot index pages for AI answers. Site owners can check their robots.txt file for Cloudflare-managed training blocks, and can test firewall-level blocks by sending requests with crawler user agents. Settings can be adjusted in the Cloudflare dashboard under AI crawler policies and AI Crawl Control. The author recommends allowing search crawlers for small business and portfolio sites so those sites can be found and cited by AI assistants.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in