Cloudflare Lets Site Owners Block AI Training Without Losing Search Indexing

·5 views

Cloudflare has launched a 'Disallow AI Training' setting that allows website owners to prevent their content from being used to train AI models without affecting search engine indexing. The feature improves on the older 'Block AI Bots' control, which could inadvertently reduce a site's search visibility by treating all AI-related crawlers the same way. Cloudflare now classifies crawlers into three categories — Search, Training, and Agent — enabling more targeted access preferences. The no-training directive is published via robots.txt through Bot Preference Sync, and major crawlers from Apple, Google, and Microsoft have committed to honoring it. Training crawlers from Amazon, Anthropic, Meta, and OpenAI are expected to be blocked under this preference, while Bingbot compliance is anticipated by early 2027.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

AI Guardrails Explained: What They Are and Why They Matter Now

AI guardrails are technical and policy mechanisms designed to limit what an AI system can do, ensuring human oversight before and after consequential actions. The concept has gained urgency as AI capabilities accelerate, with leaders from Anthropic's Dario Amodei to US House Speaker Mike Johnson publicly calling for such safeguards. Real-world failures illustrate the stakes: in 2012, Knight Capital lost $440 million in 45 minutes after unchecked trading code executed millions of unwanted transactions, ultimately destroying the firm. In a separate case, Air Canada's chatbot invented a bereavement fare policy it was never authorized to offer, and a court held the airline liable for honoring it. Across fictional and real examples alike, the recurring failure is the same — an AI agent acted without defined limits, human visibility, or a mechanism to intervene in time.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer tests Eloquent-style REST client against 62 public APIs

A Laravel developer built a REST client that mimics Eloquent's query interface and tested it against 62 public APIs, including PokéAPI and the Art Institute of Chicago API. The client allows developers to use familiar methods like paginate(), with(), and get() to interact with external REST endpoints as if they were database models. Each API's response structure, such as envelope keys and pagination styles, is handled through simple model configuration rather than custom parsing code. The tool automatically strips response envelopes, maps foreign keys to relationships, and fires concurrent requests when eager-loading related resources. The post documents real response outputs across varied API designs, highlighting how consistently the client adapted to different pagination and data-wrapping conventions.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

OpenAI Releases GPT-6 Astra With Staged Access Across ChatGPT, API, and Cloud Partners

OpenAI has launched GPT-6 Astra, described as its most capable model to date, with a phased rollout beginning with a limited group of organizations. Broader access is planned within days across ChatGPT Plus, Pro, Business, and Enterprise tiers, as well as the OpenAI API, Microsoft Azure, and AWS Bedrock. The model is built around extended reasoning, multi-agent coordination, and computer-use tasks — capabilities designed for complex, multi-step workflows rather than simple prompt-and-response interactions. Advanced cybersecurity features are being introduced more cautiously, initially restricted to trusted testers in OpenAI's Daybreak program before wider release. OpenAI has not yet published final API pricing, and availability of specific capabilities varies even where the underlying model is being rolled out.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Registrar APIs and DNS Interfaces Serve Distinct Roles in Tenant Domain Migrations

When migrating tenant subdomains on a gaming platform, registrar APIs and DNS interfaces handle fundamentally different responsibilities despite both involving domains. Registrar APIs govern domain ownership, nameserver delegation, and transfer state, while DNS interfaces manage individual zone records like A, AAAA, and CNAME entries. Mixing the two workflows risks dangerous race conditions, such as record updates succeeding in an old zone while the application already queries the new one. Best practice involves dual-publishing records in both zones, lowering TTLs in advance, and switching delegation only after verifying answers across authoritative servers, recursive resolvers, and client hostnames. Rolling back should restore delegation rather than delete tenant records, and migration state should be tracked as structured data with phase, target, and observed-answer fields.

0 comments Read more at DEV Community