Over Half of AI Crawler Requests Are Fake, Cloudflare Log Analysis Reveals

·1 views

A developer analyzing eight days of web server logs found that 991 out of 1,459 requests from self-identified AI crawlers were unverified imposters, while only 468 were legitimate. Using Cloudflare's free-tier reverse-DNS verification tool, the analysis checked whether each crawler's claimed identity matched its actual origin. Bots presenting as GPTBot were verified only 13% of the time, and all 455 Perplexity-branded requests failed verification entirely. Meta-externalagent and Applebot were the only crawlers with 100% verified traffic, making their reported numbers the only ones considered reliable. Notably, every request using the Google-Extended user agent was flagged as fake, since Google's own documentation confirms that string is never used for actual crawling.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

OAuth2 Authorization Code Flow Explained Without the Jargon

OAuth2 is an authorization framework that allows apps to access user data on third-party platforms without ever handling the user's password. Instead of credentials, apps receive a scoped access token — comparable to a valet key — granting limited, time-bound permissions on a user's behalf. For most web applications, the Authorization Code Flow is the only grant type developers need to understand, involving a four-step exchange between the browser, backend, user, and identity provider like Google or GitHub. The flow deliberately splits the process so that the sensitive client secret and final token exchange happen server-side, protecting against interception in the browser environment. A short Node.js example illustrates how the temporary authorization code received after user consent is traded server-to-server for the actual access token.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Dev's 'Smart' Notification Fix Silently Broke Alerts for Multi-Monitor Users

A developer building ShelfTalk, a real-time social app for book lovers, added a feature to suppress push notifications when users were actively viewing the app tab. The logic relied on the Page Visibility API's document.hidden property, which only returns true when a tab is minimized or fully hidden behind other windows. Users on multi-monitor setups began missing direct messages and group mentions because their ShelfTalk tab remained technically 'visible' on a second screen, even while they worked on a different app on their primary monitor. The developer discovered the flaw only after a user described their dual-screen workflow, revealing that the OS considers any on-screen window as visible regardless of user attention. The bug highlights a key limitation of the Page Visibility API: it tracks window visibility, not actual user focus or attention.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Manor AI Launches Self-Hosted Goal-Driven AI Workspace to Replace Chat-Based Tools

A team has developed Manor AI, a self-hosted AI platform built around structured, goal-driven Workspaces rather than conventional chat interfaces. Each Workspace is designed as an independent operating unit with defined goals, assigned agents, tasks, approval rules, and verifiable output artifacts. The approach aims to solve limitations of chat-based AI tools when handling long-running or multi-user workflows, where context and accountability are harder to maintain. Manor AI connects conversations, plans, knowledge bases, automations, and review processes within isolated Workspaces to prevent context bleed between unrelated projects. The source code has been published on GitHub under the Manor Sustainable Use License 1.0, which the team clarifies is source-available rather than OSI-certified open source.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why Formal Problem Specs Beat Prompt Tricks for AI-Generated Production Code

A lead strategist at Digitalizen argues that most AI-generated code failures in production stem from poorly defined problem specifications, not from model capability gaps. As reasoning models have grown more powerful, classic prompt engineering tactics like persona framing and keyword hacks have become insufficient guardrails. The author proposes 'Problem Engineering,' a structured approach that requires developers to formally define system invariants, data contracts, state mutations, fault topologies, and observability hooks before any code is generated. Without these constraints, large language models default to statistically likely but naive implementations that lack concurrency controls or robust error handling. By eliminating ambiguity upfront, developers can narrow the model's output space and consistently obtain production-ready results.

0 comments Read more at DEV Community