64% of Major News Publishers Block AI Crawlers Amid Traffic and Revenue Fears

·1 views

A review of robots.txt files from 98 major websites across four industries found that 64% of news and content publishers block at least one AI crawler, compared to just 16% of SaaS and e-commerce companies. Five outlets — The New York Times, BBC, Bloomberg, BuzzFeed, and HuffPost — have blocked all six major AI crawlers entirely, including GPTBot, ClaudeBot, and Google-Extended. The primary driver is financial: publishers are losing advertising and subscription revenue as AI-generated answers reduce the need for users to click through to original sources, with some studies linking AI Overviews to click-through-rate drops exceeding 50%. The New York Times filed a lawsuit against OpenAI and Microsoft in late 2023 over unauthorized use of its journalism, while other publishers have pursued paid licensing deals to monetize their archives. However, analysts note that blocking crawlers only prevents future citation by AI models — it does not remove content already used in model training.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer Builds AI-Powered Web App to Explore Medieval Bosnian Tombstones

A developer has created Stećak Oracle, a knowledge-driven web application focused on stećci — medieval tombstone monuments found across Bosnia and Herzegovina, Croatia, Montenegro, and Serbia. The app combines Sanity as a structured content layer, Next.js for the frontend, and Google Gemini as an AI interface to answer user questions about the monuments. Historical data — including individual monument records, motifs, inscriptions, and sources — is stored in Sanity rather than hardcoded, keeping the AI as a generation layer rather than a source of truth. To prevent misinformation, motif documents include an 'Interpretation Caution' field that distinguishes documented historical facts from interpretive claims. The project was built as an entry for the Sanity Challenge, with initial dataset sourced from the UNESCO World Heritage Centre's Stećci Medieval Tombstone Graveyards entry.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Anthropic Weighs Faster Model Launch Days After Amodei Urged AI Industry Slowdown

Anthropic CEO Dario Amodei published an essay on September 12 urging the global AI industry to voluntarily slow the rollout of advanced frontier capabilities, citing safety evaluation gaps. Just six days later, Reuters reported that Anthropic leadership is considering accelerating the release of its own next-generation model. The reported trigger is competitive pressure: OpenAI's GPT-6 Astra launch in early September gained significant enterprise and developer traction, and prospective IPO investors are questioning Anthropic's ability to hold its market position. Analysts note the apparent contradiction stems less from hypocrisy than from the structural dynamics of an uncoordinated competitive race, where unilateral restraint effectively cedes customers and revenue to rivals who continue shipping. With an IPO reportedly on the horizon, Anthropic faces pressure from institutional investors who value market share and revenue metrics over stated commitments to cautious deployment timelines.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Whoosh Library Supports Phrase and Proximity Search for Precise Text Queries

Whoosh, a pure-Python full-text search library, offers built-in phrase and proximity query support to help match terms based on word order and position. Standard bag-of-words scoring cannot capture a user's intent when they quote a phrase, so Whoosh uses term position data stored at index time to enable exact phrase matching. A quoted query like "machine learning" will only return documents where those words appear adjacently and in order, filtering out results where the terms are scattered across the text. Adding a slop value after the closing quote, such as ~2, allows a specified number of intervening words while still enforcing term order, broadening results in a controlled way. Developers can also construct Phrase queries directly in code using the Phrase class, which is useful when input comes from structured sources rather than user-typed strings.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

AI Coding Tools Boost Engineer Productivity While Quietly Eroding Core Skills

AI coding assistants have made software engineers appear more productive by accelerating output — closing tickets faster and shipping features in minutes — but this efficiency comes with a hidden cost. The deep problem-solving instincts and system-level intuition that engineers traditionally built through struggle are no longer being exercised in the same way. Productivity and cognitive development, once treated as parallel measures of engineering growth, have been decoupled by AI tools that delegate the reasoning process itself. Engineers who habitually defer to AI at the first sign of difficulty may be actively training themselves away from productive struggle, potentially degrading skills they already possess. The concern is not that AI should be abandoned, but that rising throughput metrics are being mistaken for professional growth when the two no longer reliably move together.

0 comments Read more at DEV Community