SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

AI Crawlers Quietly Drain Server Resources, Posing Hidden Costs for the Web

0
·1 views

On August 29, kernel.org administrator Konstantin Ryabitsev revealed that AI scrapers were consuming 14–16 of the site's 90 CPU cores continuously, just to render individual Git HTML pages that were already available for bulk cloning. The redundant requests highlighted how automated retrieval can impose real infrastructure costs on source websites without disrupting them visibly. AI-related web traffic spans distinct activities — model training, search indexing, and user-triggered fetches — each carrying different implications for site owners and developers. Agentic workflows can worsen the problem when multiple subagents independently fetch the same content, adding latency and server load without improving the quality of results. Experts argue that AI application developers need to optimise for distinct, relevant evidence rather than raw page volume, and should consider the server-side cost their retrieval behaviour imposes on publishers.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

How Isolating Token Services Prevents Chat Failures in Multi-Tenant Support Systems

A software architecture guide published on DEV Community outlines best practices for separating token issuance from chat data management in customer support platforms. The core recommendation is to scope each token strictly to a single tenant, conversation, actor, and capability, preventing authorization logic from becoming entangled with message history recovery. Reconnection after disconnection should rely on a durable message cursor tied to a client-device pair, not on data stored within the token service's session. The approach enforces key invariants, such as ensuring a token issued for one building or workspace can never authorize access to another, and that refreshing a token cannot inadvertently advance a client's message position. By keeping these responsibilities in separate services, the system remains resilient when any individual component degrades.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

EchidraOSS Claims Async Architecture Outperforms Cowrie and Thinkst Canary Under High Load

A technical comparison published on DEV Community evaluates how popular honeypot tools handle sudden surges in concurrent connections from botnets and internet scanners. Cowrie, built on Python's older Twisted reactor, uses a single-threaded event loop that struggles when large numbers of SSH negotiations compete for the same processing cycle. Thinkst Canary addresses the problem through dedicated hardware appliances rather than architectural improvements, making it infrastructure-grade but less flexible to deploy. EchidraOSS, by contrast, is built on Python 3.11+ with a FastAPI async core, where each connection runs as a coroutine, allowing simultaneous handling of connection floods without thread overhead or linear CPU scaling. The developers argue this design prevents the dropped connections and missed logs that undermine a honeypot's core purpose during high-traffic bursts.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

react-use-control Library Eliminates React Controlled/Uncontrolled State Boilerplate

A developer has released react-use-control, an open-source React library designed to simplify how components handle controlled and uncontrolled state. Traditional React patterns require separate value, defaultValue, and onChange props along with sync effects and mode-detection logic, which multiplies boilerplate across complex components. The library introduces a single 'control' prop per state value: passing a plain value sets an uncontrolled default, while passing a control object shares state directly with the parent, eliminating dual sources of truth. Both modes execute the same internal code path with no branching, and the library weighs in at roughly 80 lines with zero external dependencies. Benchmarks show the approach outperforms both manual implementations and Radix UI's useControllableState on controlled prop updates, and it currently powers the author's haze-ui component library.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Developer ships manual binary 4 days late after CI billing block stalls Rust project release

A developer published the v0.1.0-alpha tag for a Rust project on August 27, 2026, but attached no downloadable binary, leaving users with only a clone-and-build option. The project's CI pipeline had been billing-blocked since August 15, silently refusing all jobs with no logs, meaning the automated release workflow had never successfully run. A hand-built binary for x86_64 Linux was finally uploaded on August 31, four days and six hours after the tag was created. The developer acknowledged the release lacks DSSE attestation, was built on a personal machine rather than CI, and covers only one platform. The post reflects on how waiting for a proper pipeline effectively became an excuse for shipping nothing, which the author argues was the worse outcome despite the binary's limited provenance.

0 comments Read more at DEV Community