SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Over Half of AI Crawler Traffic on One Site Could Not Be Verified as Genuine

0
·2 views

A web publisher analyzing 13,491 crawler requests to their site over 30 days found that more than half came from AI vendors — including Meta, ByteDance, and Amazon — that publish no IP ranges or verification method, making it impossible to confirm the requests' true origin. A further 2,226 requests could not be verified because the publisher's local copy of vendor IP lists was outdated, while 4,318 requests came from addresses that did not match any published list. Notably, all 984 requests claiming to be Perplexity-User and all 103 claiming to be PerplexityBot had a 0% verification rate, while Applebot and GoogleOther scored 98% and 100% respectively. The analysis highlights that a user-agent string is simply a self-reported claim — anyone can spoof it with a single command — making unverified crawler statistics potentially unreliable. The findings point to a structural gap in AI crawler accountability, where site owners currently have no reliable way to confirm the identity of the majority of automated traffic they receive.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Axiom CLI Tool Automates Project Setup by Detecting Dependencies Automatically

A developer has released Axiom, an open-source command-line tool designed to eliminate the manual effort involved in setting up software projects. The tool automatically detects a project's required stack, dependencies, and environment configurations, then handles installation without user intervention. Axiom is available for all major operating systems and its source code is hosted on GitHub. The developer built it to address a common frustration where mismatched dependency versions and missing tools cause projects to fail before they even run. By keeping it open source, the creator aims to let the developer community add support for additional languages, frameworks, and build systems over time.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Developer details how Snaptura solves DPR scaling, sticky elements, and redaction in Chrome MV3

A developer has shared technical insights from building Snaptura, a free Chrome extension for capturing DOM elements as screenshots. The article explains how device pixel ratio (DPR) must be factored into every element capture to avoid blurry or misaligned crops on high-resolution displays. It also covers how sticky and fixed-position elements are handled during full-page scroll-and-stitch captures to prevent repeated headers from appearing across folds. For privacy, the extension redacts sensitive content at the DOM level before rasterization, ensuring secret data is never written into pixels. Additional challenges discussed include managing Chrome MV3 service worker lifecycle limitations and offloading heavy tasks like GIF encoding to offscreen documents.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Developer Builds Fully Operational Home Security Operations Center With Custom AI

A cybersecurity professional has documented the construction of an enterprise-grade Security Operations Center (SOC) running entirely from his home, using open-source tools including Wazuh, Zeek, Suricata, and Elastic. Unlike typical home lab projects, the system runs continuously, self-monitors via a dead-man's switch, and uses tiered triage to filter out roughly 99% of alert noise before notifying the operator. A custom AI triage engine — built from scratch and combining deterministic security logic with a locally-run language model — analyzes threats without sending any data to external cloud services. The system also prioritizes remediation by highlighting vulnerabilities actively being exploited in the wild rather than generating generic vulnerability lists. The builder, who runs cybersecurity firm ByTE X Bit Technologies LLC, is publishing the full architecture and methodology as a multi-part public series.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Developer Builds AI-Assisted Interactive 3D Science Museum with Six Themed Exhibits

A developer has built Loupe, an interactive digital museum featuring six science exhibits covering topics such as space, dinosaurs, human evolution, and the Apollo 11 lunar descent. Each exhibit uses a distinct interaction model — from scrollable mission narratives to explorable 3D planetary surfaces — designed to match its subject matter. The project was built using a stack that includes Next.js, React Three Fiber, Three.js, GSAP, and Zustand, with AI tools Claude and Codex assisting throughout development. The application separates content, logic, and components into distinct layers to manage the complexity of interactive 3D experiences in the browser. The developer shared the project and its architecture on DEV Community to illustrate how AI coding agents can support ambitious, non-standard web applications beyond basic CRUD functionality.

0 comments Read more at DEV Community