Cloudflare silently blocked ChatGPT, Perplexity and Claude bots for three weeks
A developer discovered that Cloudflare's 'Block AI Bots' security setting had been silently returning HTTP 403 errors to AI search crawlers — including OpenAI's SearchBot, PerplexityBot, and Claude-SearchBot — for three weeks, while standard search engines like Googlebot passed through unaffected. Because the blocks occurred at the edge before requests reached the origin server, Apache access logs recorded nothing, making the problem invisible to conventional monitoring tools. The developer only uncovered the issue by manually simulating each bot's user agent with curl against the live public URL, bypassing local network shortcuts. A secondary problem was also found: fail2ban was misreading Cloudflare relay IPs as malicious clients and banning them, which periodically caused HTTP 521 errors for all visitors. The developer responded by building a daily automated crawler check that probes the real public URL across 16 user agents and alerts on any unexpected status change.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in