Script compares Googlebot crawl logs per site section to spot post-deploy drops
A DEV Community guide explains how to detect which website sections Googlebot crawled less after a deployment. Instead of relying on Search Console's delayed sitewide graph, it uses raw access logs for a precise per-section view. The provided Python script filters Googlebot requests from Nginx or Apache logs, groups URLs by first path segment, and compares daily crawl counts and status codes before and after a cutoff time. Results are sorted by the largest crawl drops, with the status mix shown for each period. The article interprets patterns such as lower crawl rates from reduced internal links, rising redirects after URL changes, or new server errors that need investigation.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in