Cloudflare's AI Bot Rules Silently Block Googlebot, Bypassing robots.txt

·1 views

A September 2026 Cloudflare policy change altered default settings for AI-related crawlers, causing Googlebot, Bingbot, and Applebot to receive 403 errors on ad-supported pages across new and free-tier accounts. Because Cloudflare classifies Googlebot as a mixed-use crawler serving both search and AI training, it falls under the more restrictive AI training block when that setting is enabled. The problem goes undetected because robots.txt still shows an open allow directive, while the actual block is enforced at the CDN edge layer before the origin server is ever reached. Cloudflare's own data shows Googlebot's crawl share dropped to 27.49% in Q2 2026, down sharply from 57.20% a year prior. Site owners can diagnose the issue by comparing HTTP response codes for browser versus Googlebot user-agent requests, or by using Google Search Console's URL Inspection tool for verified confirmation.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How Mojo Uses SIMD to Speed Up Text Search Across Large Documents

SIMD, or Single Instruction Multiple Data, allows a processor to compare multiple bytes simultaneously rather than one at a time, making it far more efficient for large-scale text searches. The Mojo programming language exposes this CPU-level capability through a typed SIMD construct where both the data type and number of lanes are part of the value's type signature. A tutorial on DEV Community walks through building a Mojo application called ResearchLens, which uses SIMD to quickly identify candidate word matches in research documents. The approach pairs a fast SIMD scan for first-byte matches with a scalar verifier to confirm full word matches and handle edge cases like partial vectors. No GPU is required, as the SIMD hardware used is already present in modern CPUs.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why HPC Clusters Use InfiniBand: Low Latency and RDMA Explained

High-performance computing (HPC) clusters often deploy InfiniBand alongside standard Ethernet because tightly coupled workloads demand more than just high bandwidth. Applications running across dozens or hundreds of nodes continuously exchange data, making both latency and throughput critical to overall performance. InfiniBand's key advantage lies in Remote Direct Memory Access (RDMA), which enables data transfers directly between nodes' memory with minimal CPU and operating system involvement, reducing communication overhead. However, InfiniBand is not the only option — high-speed Ethernet and RoCE, which brings RDMA capabilities to Ethernet, are also used depending on workload requirements. Ultimately, the network is as vital to HPC performance as the CPUs themselves, since powerful processors are ineffective if nodes spend excessive time waiting for data.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

WebDecoy Plugin Lets Fastify APIs Detect Scrapers and Bots in Monitor Mode

Developers can add request detection to Fastify APIs using the open-source WebDecoy plugin, which identifies whether traffic comes from frontends, integrations, or scrapers hitting the same endpoint. The plugin operates in a non-blocking monitor mode, meaning suspicious requests are flagged and logged but not blocked, leaving API responses unchanged. A tripwire rule can be configured on sensitive paths such as /.env to trigger a DENY verdict whenever those routes are probed. Detection results are logged locally by default, and connecting an API key unlocks a cloud dashboard where flagged requests appear as labeled records. The walkthrough targets Node.js 22.12 or later and uses Fastify 5 alongside the @webdecoy/fastify and @webdecoy/node packages.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

AfterImage v1.5 syncs live F1 session states with photo archive using a shared clock

Developer building AfterImage, an independent F1 archive using TypeScript, React, Next.js, and Supabase, has detailed the approach taken in version 1.5 to synchronise live race timing with a photo archive. The update introduces a race-weekend hub that separates scheduled session windows from provider-confirmed live states, preventing the interface from falsely marking a session as live due to delays. Browser clients use server timestamps interpolated with performance.now() rather than device clocks, and a polling fallback ensures a slow response cannot roll displayed state backward. Photographs carry optional metadata fields such as capture time and circuit corner, which are never guessed or substituted with upload time. The project also maintains clear attribution by keeping photographer credits and upload ownership as distinct fields in the photo model.

0 comments Read more at DEV Community