Most of my server's 'bot' traffic was its own health checks, not robots
The author added a counter to their own site's analytics to see who was visiting and found an inflated figure of 21,791 visits. Grouping the data showed that 20,709 of those came from curl requests to the homepage, sent from three addresses, which turned out to be each app container's health check hitting its own server. Link previews from Facebook also contaminated the numbers, and the real AI crawlers had only sent one request each. The author fixed the counting logic, removed the heartbeat traffic, and found that the actual figure for the first day was 914 visits from 26 addresses. They also noted that well-behaved crawlers like Googlebot, Microsoft and OpenAI self-identify through the From: header, which helped answer who runs the bots.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.


Discussion (0)
Log in to join the discussion and vote.
Log in