Developer opens site to all AI crawlers, then realizes he cannot measure real traffic

·1 views

Marco Bellingeri, a cloud and security engineer, launched a personal website on 5 July and deliberately allowed all AI crawlers unrestricted access via robots.txt, hoping to gain visibility rather than guard his content. After Cloudflare reported 33,561 pageviews in the first month, he realized the figure was meaningless because the platform's free plan cannot distinguish human visitors from bots. Testing AI tools revealed an uneven outcome: Perplexity cited him first for his name, while ChatGPT ignored him entirely and reframed the query as a topic search. To investigate whether crawlers were actually visiting, he built a lightweight logging system inside a Cloudflare Worker that records only HTML requests and identifies bots by User-Agent, without storing any personal data. The exercise exposed a key diagnostic question: if crawlers never arrive, the problem is access; if they arrive but do not cite, the problem is content extractability.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How Random Forests Cut Variance: The Math Behind Bagging and Tree Averaging

Random forests reduce prediction variance by averaging many deep, unpruned decision trees, each trained on a different bootstrap sample of the data. Random feature selection at every split ensures trees remain diverse, preventing a single dominant feature from making all trees look alike. A mathematical identity shows that the mean squared error of an ensemble always equals the average individual tree error minus the spread among trees, explaining why diversity directly drives accuracy gains. Each tree leaves out roughly 37% of training rows, enabling out-of-bag error estimation as a free, honest validation method without a separate holdout set. Unlike boosting, adding more trees to a random forest converges to an error floor and cannot overfit, making the two methods fundamentally different in how they use depth, data, and sequential dependence.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

OpenClaw vs Hermes Agent: Two Rival Open Source AI Frameworks Compared

OpenClaw and Hermes Agent have emerged as two of the most discussed open source autonomous agent frameworks in 2026, each taking a fundamentally different approach to AI task automation. OpenClaw treats agents as a team of workers, using Markdown-based identity files to define each agent's role, memory, and behavior, making it suited for structured, multi-agent workflows. Hermes Agent, released by Nous Research in February 2026, focuses on a single self-improving agent that automatically distills complex tasks into reusable skills and refines them over time. Hermes gained over 215,000 GitHub stars within weeks of launch, making it one of the fastest-growing agent projects of the year. Developers are advised to choose OpenClaw for team-oriented, deterministic setups and Hermes for a continuously learning, solo agent experience.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Rust API Design: How to Manage Type Changes Without Breaking User Code

A technical guide on advanced Rust API design warns developers to think carefully before making interface changes visible to users, as frequent backward-incompatible updates frustrate downstream consumers. The article explains that even subtle modifications—like adding a field to a public struct—can silently break existing user code that previously compiled without issue. To minimize this risk, developers are advised to use Rust's visibility modifiers such as pub(crate) and pub(in path) to limit how much of an API is publicly exposed. The fewer public types an API surfaces, the greater the freedom a developer retains to make internal changes later. The guide also introduces the non_exhaustive attribute as a tool to signal that types may grow over time, helping users write more future-proof code.

0 comments Read more at DEV Community

ProgrammingHacker News ·

UK Met Office Launches Glacier Tracking Tool on Climate Dashboard

The UK Met Office has added a glacier monitoring section to its online Climate Dashboard. The tool provides data visualizations tracking the state and changes of glaciers globally. Glaciers are key indicators of climate change, making their inclusion on such platforms scientifically significant. The dashboard is publicly accessible and aims to present climate data in a transparent, understandable format.

0 comments Read more at Hacker News