Developer finds stale CDN cache silently blocked Google from indexing his site for 4 days
A developer running a technical SEO audit on his own static portfolio site discovered that a stale CDN-cached robots.txt file had been instructing all crawlers, including Googlebot, to disallow the entire site. The cached file, served by Surge.sh's CDN, was nearly 10 hours old and overriding a correctly deployed version that allowed crawling. No errors, console warnings, or performance drops indicated the problem, making it effectively invisible without a manual audit. The developer initially mistook the issue as resolved after using a cache-busting query string, before realising Googlebot always requests the bare /robots.txt path with no parameters. He published the full audit report as a free sample to illustrate how critical indexability bugs can go undetected without reading raw server responses the way a search engine crawler would.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in