Cloudflare Adds One-Click Tool to Block AI Scrapers While Preserving SEO

·1 views

Cloudflare introduced a new bot category called 'Accountable Mixed-Use AI Crawlers' in June 2024, enabling website owners to block AI data-harvesting bots with a single click. Unlike traditional search engine crawlers such as Googlebot and Bingbot, AI crawlers often ignore robots.txt rules and can send hundreds of requests per second to extract text, images, and metadata for model training. Cloudflare's Firewall Tools panel allows users to set a default 'Block' action for this category while whitelisting legitimate search bots to protect organic search rankings. Website owners can also write custom firewall rules targeting specific user agents like GPTBot, Claude, or Bard, or opt for a softer JavaScript Challenge approach instead of an outright block. The move aligns with growing legal pressure under the EU's GDPR and DSM Directive, as well as the US DMCA, both of which impose penalties for unauthorized reuse of protected content.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How to Build a Durable Node.js Service for Scanned Claims Processing

A Node.js intake service for scanned claims should accept bounded uploads, validate them before queuing, and store original files immutably as the source of truth. Rendering should happen asynchronously via separate workers, not during the HTTP request, to avoid memory exhaustion under burst traffic. Developers are advised to separate queues by failure domain — intake, render, and notification queues should operate independently to prevent one bottleneck from cascading. Each job should carry a SHA-256 digest to ensure idempotency, making duplicate deliveries harmless through a uniqueness constraint on claim ID, digest, and operation. A state machine tracking transitions like accepted, rendering, complete, and rejected is recommended over relying solely on queue status for operational visibility.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

S3-Compatible Storage Has Hidden Pitfalls That Break Apps in Production

Amazon S3's API became an industry standard, prompting many object-storage providers to market themselves as 'S3-compatible' — but compatibility covers API verbs, not all edge-case behaviors. A developer migrating the app StoryKept to a cheaper storage provider discovered that presigned URLs silently failed in production because the provider used path-style addressing while the SDK defaulted to virtual-hosted style, causing signature mismatches. The issue passed all local tests and only surfaced when real users attempted to fetch files, since tests typically check URL generation rather than full round-trips. A second problem arose from missing CORS configuration, which caused browsers to block media requests entirely even after the URL signing issue was resolved. Both failures highlight that switching S3-compatible providers requires careful attention to addressing style settings and cross-origin policies, not just endpoint credentials.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Understanding the N+1 Database Query Problem and How to Fix It

The N+1 problem is a common database performance issue where an application runs one initial query to fetch a list of records, then executes an additional query for each individual record in that list. For example, fetching 100 users and then loading each user's posts separately results in 101 total database queries instead of just one. At small scale this may seem harmless, but as data grows to thousands or lakhs of records, the cumulative query time can make an application unacceptably slow. The standard solution is eager loading using SQL JOINs, which retrieves all related data in a single query; ORMs like Sequelize and Prisma offer built-in support for this via their 'include' options. Because the problem only becomes visible under high data volume, developers are advised to enable query logging and watch for multiple similar queries triggered by a single request.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Solo Dev's SmartConvert Hits 45 Users and AdSense Rejection in Week Two

A solo developer building SmartConvert, a small online tools website, recorded just 45 users, 340 search impressions, and 4 search clicks in its latest weekly update. Google rejected the site's AdSense application, citing low-value content — an outcome the developer said was expected given the site's limited tools and minimal search traffic. Three new PDF tools were added last week, but the developer has chosen to pause further development to allow Google time to index the new pages. The AdSense rejection prompted a broader reflection on the gap between launching a website and building one that earns traffic and revenue. The developer concluded that improving the product's quality and usefulness is now a higher priority than adding new features.

0 comments Read more at DEV Community