How to Detect Crawlers on a Static Astro Site Using Cloudflare Workers

·1 views

Static Astro sites prerender HTML, meaning crawlers can download pages without triggering browser-based analytics scripts, leaving automated traffic largely invisible to site owners. A developer guide explains how to deploy WebDecoy's edge sensor via Cloudflare Workers to observe all incoming requests — including those from bots that never execute JavaScript. The setup requires a custom domain proxied through Cloudflare and does not need changes to Astro's server rendering configuration or build process. Once installed, the Worker intercepts requests at the network edge before they reach the origin, logging detections in the WebDecoy dashboard. Owners can verify the sensor is working by sending a test curl request with a reserved user-agent string and confirming the labeled detection appears in the dashboard.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How Mojo Uses SIMD to Speed Up Text Search Across Large Documents

SIMD, or Single Instruction Multiple Data, allows a processor to compare multiple bytes simultaneously rather than one at a time, making it far more efficient for large-scale text searches. The Mojo programming language exposes this CPU-level capability through a typed SIMD construct where both the data type and number of lanes are part of the value's type signature. A tutorial on DEV Community walks through building a Mojo application called ResearchLens, which uses SIMD to quickly identify candidate word matches in research documents. The approach pairs a fast SIMD scan for first-byte matches with a scalar verifier to confirm full word matches and handle edge cases like partial vectors. No GPU is required, as the SIMD hardware used is already present in modern CPUs.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why HPC Clusters Use InfiniBand: Low Latency and RDMA Explained

High-performance computing (HPC) clusters often deploy InfiniBand alongside standard Ethernet because tightly coupled workloads demand more than just high bandwidth. Applications running across dozens or hundreds of nodes continuously exchange data, making both latency and throughput critical to overall performance. InfiniBand's key advantage lies in Remote Direct Memory Access (RDMA), which enables data transfers directly between nodes' memory with minimal CPU and operating system involvement, reducing communication overhead. However, InfiniBand is not the only option — high-speed Ethernet and RoCE, which brings RDMA capabilities to Ethernet, are also used depending on workload requirements. Ultimately, the network is as vital to HPC performance as the CPUs themselves, since powerful processors are ineffective if nodes spend excessive time waiting for data.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

WebDecoy Plugin Lets Fastify APIs Detect Scrapers and Bots in Monitor Mode

Developers can add request detection to Fastify APIs using the open-source WebDecoy plugin, which identifies whether traffic comes from frontends, integrations, or scrapers hitting the same endpoint. The plugin operates in a non-blocking monitor mode, meaning suspicious requests are flagged and logged but not blocked, leaving API responses unchanged. A tripwire rule can be configured on sensitive paths such as /.env to trigger a DENY verdict whenever those routes are probed. Detection results are logged locally by default, and connecting an API key unlocks a cloud dashboard where flagged requests appear as labeled records. The walkthrough targets Node.js 22.12 or later and uses Fastify 5 alongside the @webdecoy/fastify and @webdecoy/node packages.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Cloudflare's AI Bot Rules Silently Block Googlebot, Bypassing robots.txt

A September 2026 Cloudflare policy change altered default settings for AI-related crawlers, causing Googlebot, Bingbot, and Applebot to receive 403 errors on ad-supported pages across new and free-tier accounts. Because Cloudflare classifies Googlebot as a mixed-use crawler serving both search and AI training, it falls under the more restrictive AI training block when that setting is enabled. The problem goes undetected because robots.txt still shows an open allow directive, while the actual block is enforced at the CDN edge layer before the origin server is ever reached. Cloudflare's own data shows Googlebot's crawl share dropped to 27.49% in Q2 2026, down sharply from 57.20% a year prior. Site owners can diagnose the issue by comparing HTTP response codes for browser versus Googlebot user-agent requests, or by using Google Search Console's URL Inspection tool for verified confirmation.

0 comments Read more at DEV Community