Websites blocking AI crawlers largely overlook answer-engine bots

A new analysis of robots.txt configurations reveals that many websites actively block AI training crawlers but fail to restrict answer-time bots that retrieve content in real time. This inconsistency suggests site owners may not fully understand the distinction between crawlers that harvest data for model training versus those that fetch content to answer user queries directly. The oversight could mean that even sites trying to protect their content are still being accessed and used by AI-powered answer engines. The findings highlight a growing gap in awareness around the different types of automated web access as AI tools proliferate.