Why robots.txt Alone Can't Keep a Staging Site Out of Google

Many staging sites get indexed after developers rely only on a Disallow rule in robots.txt, which blocks crawling but not indexing. Google can list a blocked URL it discovers through external links, and the blocked page's noindex tag goes unseen. The public robots.txt file can also expose sensitive paths to attackers. The author recommends requiring authentication such as HTTP Basic Auth or an IP allowlist, plus an X-Robots-Tag noindex header as a second layer. Staging hostnames also leak through Certificate Transparency logs, shared sitemaps, referrers, and hard-coded links.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in