SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Nearly Half of Top News Sites Now Block OpenAI's Web Crawler, Reuters Institute Finds

0
·2 views

Major news publishers including The New York Times, CNN, ABC, and the Guardian have blocked OpenAI's GPTBot crawler, reflecting a broader industry shift in how journalism is treated as AI training material. A Reuters Institute study found that by late 2023, approximately 48% of leading news sites across ten countries had restricted OpenAI's crawlers, with legacy outlets more likely to block than newer ones. Publishers are using robots.txt files to control which AI bots can access their content, with some applying restrictions beyond just GPTBot to wider categories of AI crawlers. The move gives publishers greater leverage to negotiate licensing deals — as Axel Springer has reportedly pursued — rather than allowing unrestricted access to their archives. As premium news content becomes harder to crawl freely, AI developers face growing pressure to secure formal data agreements or find alternative sources for high-quality training data.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Why AI Systems Fail: The Real Problem Often Lies in Human-Built Data Design

A technical analysis argues that AI failures are frequently rooted in poorly defined data workflows and unclear success criteria built by humans before any model is deployed. The piece contends that every automated system inherits the blind spots of its designers, particularly when data shapes, missing values, and edge cases have not been explicitly mapped out. Before deploying an AI model, developers are urged to document each data transformation stage and define observable, agreed-upon conditions for acceptable and unacceptable outputs. Human reviewers must first demonstrate consistent judgment on these distinctions, since measuring a model against internal human disagreement produces unreliable results. The article also highlights mutation testing as a practical tool to assess how sensitive a test suite is to small faults, helping teams identify weaknesses in their safety contracts rather than reflexively blaming the AI.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Solo developer builds AI resume-scoring and job-matching agent using Gemini and FastAPI

A solo developer spent several months building Reclaim, an AI-powered job-search tool for engineers, working nights and weekends. The platform uses Google's Gemini AI to parse resumes, score them for substantiated claims rather than keyword density, and match candidates against real open roles. The tech stack includes Next.js, FastAPI, Supabase, Clerk for authentication, and Stripe for payments, with Gemini model tiers selected based on task complexity to control costs. Key engineering challenges included a webhook signature failure caused by a trailing newline in an environment variable, a race condition creating duplicate user records, and a silent API cost leak from unentitled users triggering paid AI calls. The tool is live at reclaim.careers and offers a free resume scan without requiring a sign-up.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Yelp Signs Licensing Deal with OpenAI to Bring Reviews and Business Data to ChatGPT

Yelp has confirmed a licensing agreement with OpenAI, announced as part of Yelp's February 2026 earnings release, that will integrate its reviews, ratings, photos, and business information into ChatGPT's responses to local queries. The deal extends Yelp's data-licensing strategy beyond traditional search platforms and into AI-driven local discovery ecosystems. ChatGPT users asking about nearby restaurants, contractors, or businesses are expected to see Yelp content surfaced directly in conversational responses. Yelp has also signaled a near-term integration of its Request a Quote feature, which would allow users to initiate contact with local service providers from within the ChatGPT interface. The agreement underscores the growing commercial value of licensed local data and signals that business visibility is increasingly shaped by third-party AI platforms, not just conventional search listings.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Five Essential WordPress Plugins New Sites Need and What to Avoid

New WordPress website owners often make the mistake of installing too many plugins, when only five or six are typically necessary for a functional, secure site. Experts warn that plugins account for the vast majority of WordPress vulnerabilities, and unnecessary or poorly coded ones can cause slowdowns, conflicts, and security risks. The recommended essentials include one SEO plugin such as Rank Math or Yoast, a contact form tool like WPForms Lite, a security plugin such as Wordfence, a backup solution like UpdraftPlus, and a caching plugin suited to the hosting environment. The guiding principle is one plugin per task, avoiding duplicates and removing anything that does not solve a clear, immediate need. Keeping the plugin list lean makes a website easier to maintain, faster to update, and less prone to compatibility issues.

0 comments Read more at DEV Community