Developer's Dashboard Upgrade Reveals Unverified Crawlers Target Non-Existent Site Paths
A web developer running a small Japanese-language site about life in Davao, Philippines, updated his crawler-tracking plugin after a reader flagged a flaw in how unverified bot requests were being counted. The fix separated requests that failed verification from those never checked at all, and added per-path verification data rather than reporting only crawler-level totals. Analysis of the seven days ending September 22 showed that every verified check on real, existing pages matched the crawler's claimed identity. However, all checks on three non-existent paths — including a .env credentials file and two software-specific API endpoints — failed verification, with ClaudeBot listed as the top visitor for each. Across the full week, 94% of ClaudeBot's 599 total requests to the site did verify as originating from Anthropic's published IP ranges, which the company began releasing on August 1.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in