Cloudflare's AI Crawler Policy: How to Block Scrapers Without Hurting SEO

·27 views

Cloudflare introduced its 'Accountable Mixed-Use AI Crawlers' policy in March 2024, automatically blocking crawlers that do not declare their purpose via HTTP headers. The policy has gained traction amid rising data-privacy regulations such as GDPR and CCPA, which carry fines of up to 4% of annual revenue for misuse of data. Platforms including Medium, Substack, and YouTube have reported a 73% surge in content-removal requests linked to AI training activity. Website owners can protect their content by configuring robots.txt rules, adding custom firewall rules in the Cloudflare dashboard, and enabling JavaScript challenges for sensitive pages. The key challenge is distinguishing legitimate search-engine bots like Googlebot and Bingbot from unauthorized AI scrapers to avoid unintended drops in organic search traffic.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Terence Tao Outlines Six Essential Mathematical Ideas

Mathematician Terence Tao has published an online piece titled 'Six Math Essentials' on his personal website. The article presents a set of core mathematical concepts that Tao considers foundational. The post was shared on Hacker News, where it gained 11 points and one comment. The provided source contains only the title, link, and engagement metrics, so the specific content of the six essentials is not described here.

0 comments Read more at Hacker News

ProgrammingHacker News ·

Sherman Anti-Trust Act of 1890: Archival Record of the Landmark US Competition Law

This is a link to the National Archives page on the Sherman Anti-Trust Act, a milestone federal document from 1890. The page provides access to the historical text of the legislation. The item was shared on Hacker News, where it drew 18 points and 3 comments. The source material offers only the document link and discussion metadata, without additional analysis or context in the provided text.

0 comments Read more at Hacker News

ProgrammingDEV Community ·

Scaffoldr Proposes Product-Based Project Starters Over Stack-Based Scaffolding

The author argues that existing tools like create-next-app, create-t3-app, and SaaS starter kits organize project setup around technology stacks or fixed product templates. Modular scaffolders ask which technologies to use, while starter kits lock developers into someone else's choices, which can drift over time. The proposed Scaffoldr tool would instead let developers specify a product type, such as ticketing, with swappable modules for payments, databases, auth, and email. The author notes that African payment providers like Paystack are often overlooked, and the project is not yet built, with maintenance and developer interest remaining open challenges.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Yonto Player moves its source contract into JavaScript plugins on QuickJS

The Yonto Player TV app connects to media servers and folders through a single ContentSourceAdapter interface. Its source contract began as an HTTP API backed by a hosted service, but that required someone to run and update a middle layer whenever a media server changed its fields. The developer kept the contract's JSON shapes as the single definition and made transport an implementation detail, allowing sources to run as JavaScript plugins inside the app. Each plugin is one file whose manifest, declaring its identity and permitted hosts, is read as plain text, and it exposes required functions such as search and getMediaList. Plugins reach the outside world only through a restricted global object, which limits them to requests, configuration, a small cache, logging and parsing helpers.

0 comments Read more at DEV Community