Sitemap and robots.txt audit finds a trailing-slash rule that blocks nothing
A developer reviewed the sitemap of the Notifio app, which is partly generated from route arrays and partly hand-maintained. Counting routes showed 38 sitemap URLs against 40 served page URLs, with the difference traced to two pages intentionally left out. One of those, the maintenance page, carries noindex and redirects when inactive, so it needs no action. The post-checkout success page is excluded in robots.txt, but the disallow rule includes a trailing slash, and because the page is served without one, the rule never matches it. The author argues that blocking crawler access is counterproductive here, since a noindex tag only works if crawlers can fetch the page, and recommends removing that rule instead of adding the missing slash.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.


Discussion (0)
Log in to join the discussion and vote.
Log in