WP Engine Blocks AI Crawlers at Platform Level, Bypassing WordPress Controls
A documented case involving WP Engine reveals that the managed WordPress host applies platform-level restrictions that block or rate-limit major AI crawlers — including GPTBot, ClaudeBot, and PerplexityBot — before requests ever reach a customer's WordPress installation. The restriction is reportedly non-disableable and invisible to standard WordPress settings, robots.txt, or typical SEO dashboards. Search Engine Land's investigation, supported by Cloudflare logs and bot-specific curl probes, found that a site can appear healthy in conventional SEO tools while remaining completely inaccessible to AI crawlers. This exposes a critical blind spot for teams working on generative engine optimization (GEO), as traditional indexing checks and traffic analytics do not verify AI crawler fetchability. Experts now recommend that GEO audits include explicit bot-level fetch tests to confirm whether AI systems can actually retrieve key pages.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in