SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Nearly Half of Top News Sites Now Block OpenAI's Crawler, Reuters Institute Finds

0
·5 views

Major news publishers are increasingly restricting AI web crawlers, with a Reuters Institute study finding that about 48% of the world's top online news sites had blocked OpenAI's GPTBot by end of 2023. The response was sharpest in the United States, where roughly 79% of leading news sites blocked OpenAI's crawler, compared to only 24% blocking Google's AI crawler globally. Publishers are concerned that their reporting may be used to train AI models or generate answers without proper compensation, attribution, or referral traffic. High-profile actions include The Guardian blocking GPTBot in September 2023 and The New York Times suing OpenAI and Microsoft in December 2023 over alleged misuse of its content. While some publishers opt for outright blocking, others like Axel Springer have pursued licensing deals, reflecting an evolving and unsettled commercial landscape for AI access to journalism.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Apple Sign-In Breaks Every 6 Months Due to Expiring JWT Client Secret

Developers using Sign in with Apple on the web may find authentication suddenly failing with an 'invalid_client' error approximately six months after setup, caused by Apple's mandatory expiry limit on JWT client secrets. Unlike other OAuth providers such as Google or GitHub, Apple requires developers to manually generate a time-limited JWT signed with a private .p8 key, which expires after a maximum of roughly 15.7 million seconds. Apple sends no warning when the secret expires, meaning users typically encounter the failure before developers do. The standard fix involves generating a fresh JWT and updating it in the authentication provider's dashboard, but this must be repeated every six months. To automate the process, a GitHub Action called apple-client-secret-rotator has been released, which regenerates and updates the secret on a scheduled basis, eliminating the need for manual intervention.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Build Only What Differentiates Your MVP, Rent Everything Else

When scoping a minimum viable product, founders face a critical early decision: what to build themselves and what to purchase as a service. The core principle is that only the unique, differentiating feature of a product warrants custom development, while common infrastructure like authentication, payments, and email should be rented from specialist providers. Building commodity capabilities in-house wastes time that could be spent on the actual value proposition, and specialist vendors bring security, compliance, and reliability that small teams cannot replicate. However, renting carries its own risks — vendor lock-in, price changes, and API instability — so dependencies should be isolated behind internal interfaces to limit exposure. The most dangerous trap is mistaking a capability that sits adjacent to the core differentiator for part of it, leading teams to over-build where off-the-shelf solutions already exist.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

How an AI customer service system quietly leaked personal data through overlooked paths

A developer building an agentic AI customer service system discovered that personal data could escape through three distinct boundaries, not one as originally assumed. While intake tokenization was applied to inbound customer emails, CRM tool results containing names, emails, and phone numbers were returned directly into the model's context window without any sanitization. Automated tests consistently passed because they only monitored the model's output — the least dangerous of the three data boundaries — leaving the return path from internal systems entirely uninstrumented. The system also grew unexpectedly from two runtime containers to twenty-seven services, with most of that complexity driven by unplanned requirements rather than new features. The author concludes that a copilot assisting a human agent can tolerate these gaps, but a fully autonomous customer service system cannot.

0 comments Read more at DEV Community