Cloudflare Offers Tool to Block AI Training Crawlers Without Hiding from Search

·1 views

Cloudflare has introduced a new feature allowing website owners to block AI training crawlers while remaining visible to traditional search engines. The tool addresses a growing concern among publishers who want to prevent their content from being used to train AI models without losing search discoverability. Cloudflare distinguishes between crawlers used for AI training purposes and those used for standard search indexing. The feature gives site owners more granular control over how their content is accessed and used by different types of automated bots.

Read the full story at Hacker News

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How to Build a Durable Node.js Service for Scanned Claims Processing

A Node.js intake service for scanned claims should accept bounded uploads, validate them before queuing, and store original files immutably as the source of truth. Rendering should happen asynchronously via separate workers, not during the HTTP request, to avoid memory exhaustion under burst traffic. Developers are advised to separate queues by failure domain — intake, render, and notification queues should operate independently to prevent one bottleneck from cascading. Each job should carry a SHA-256 digest to ensure idempotency, making duplicate deliveries harmless through a uniqueness constraint on claim ID, digest, and operation. A state machine tracking transitions like accepted, rendering, complete, and rejected is recommended over relying solely on queue status for operational visibility.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

S3-Compatible Storage Has Hidden Pitfalls That Break Apps in Production

Amazon S3's API became an industry standard, prompting many object-storage providers to market themselves as 'S3-compatible' — but compatibility covers API verbs, not all edge-case behaviors. A developer migrating the app StoryKept to a cheaper storage provider discovered that presigned URLs silently failed in production because the provider used path-style addressing while the SDK defaulted to virtual-hosted style, causing signature mismatches. The issue passed all local tests and only surfaced when real users attempted to fetch files, since tests typically check URL generation rather than full round-trips. A second problem arose from missing CORS configuration, which caused browsers to block media requests entirely even after the URL signing issue was resolved. Both failures highlight that switching S3-compatible providers requires careful attention to addressing style settings and cross-origin policies, not just endpoint credentials.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Understanding the N+1 Database Query Problem and How to Fix It

The N+1 problem is a common database performance issue where an application runs one initial query to fetch a list of records, then executes an additional query for each individual record in that list. For example, fetching 100 users and then loading each user's posts separately results in 101 total database queries instead of just one. At small scale this may seem harmless, but as data grows to thousands or lakhs of records, the cumulative query time can make an application unacceptably slow. The standard solution is eager loading using SQL JOINs, which retrieves all related data in a single query; ORMs like Sequelize and Prisma offer built-in support for this via their 'include' options. Because the problem only becomes visible under high data volume, developers are advised to enable query logging and watch for multiple similar queries triggered by a single request.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Solo Dev's SmartConvert Hits 45 Users and AdSense Rejection in Week Two

A solo developer building SmartConvert, a small online tools website, recorded just 45 users, 340 search impressions, and 4 search clicks in its latest weekly update. Google rejected the site's AdSense application, citing low-value content — an outcome the developer said was expected given the site's limited tools and minimal search traffic. Three new PDF tools were added last week, but the developer has chosen to pause further development to allow Google time to index the new pages. The AdSense rejection prompted a broader reflection on the gap between launching a website and building one that earns traffic and revenue. The developer concluded that improving the product's quality and usefulness is now a higher priority than adding new features.

0 comments Read more at DEV Community