SShortSingh.
Back to feed

Cloudflare Defaults May Block ChatGPT Training Crawlers on Your Site

0
·4 views

A developer audit of 286 portfolio sites found that 19 blocked GPTBot and ClaudeBot, and 17 of those blocks were added by Cloudflare's dashboard rather than by site owners. The article explains that AI crawlers serve different purposes: training crawlers like GPTBot and ClaudeBot collect content for future models, while search and retrieval crawlers like OAI-SearchBot and PerplexityBot index pages for AI answers. Site owners can check their robots.txt file for Cloudflare-managed training blocks, and can test firewall-level blocks by sending requests with crawler user agents. Settings can be adjusted in the Cloudflare dashboard under AI crawler policies and AI Crawl Control. The author recommends allowing search crawlers for small business and portfolio sites so those sites can be found and cited by AI assistants.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Five Concepts That Make AI Coding Agents Work Better

AI coding agents do more than autocomplete, since they read files, run commands, review output, and choose next steps. Many failures, such as forgotten instructions or false claims of completion, stem from misunderstanding how these agents operate rather than from the model itself. A key limitation is that agents only process what fits in their current context, so long conversations, large files, and vague tasks can cause important details to be lost. The article recommends having agents plan before writing code, reviewing AI-generated code rather than accepting it blindly, and avoiding repeated instructions by saving them for future sessions. It also advises giving agents access to real tools instead of limiting them to chat.

0
ProgrammingDEV Community ·

REDCap CVE-2026-90817 Lets Unauthenticated Attackers Execute Code via Public Surveys

A critical flaw tracked as CVE-2026-90817 in REDCap's Data Import handling allows code execution when a file parameter is chained across several trust boundaries. The attack can begin through a public survey, where a double-encoded parameter leads to a user-rights state that grants super-user privileges without authentication. The import routine trusts a client-selected path, rewrites the resource, and writes it to a predictable cache file that is later included as PHP. Affected versions start at 13.3.0 and are fixed in 16.0.49 LTS, 17.3.10 LTS, and 17.4.4, with CISA confirming known exploitation and a November 21 federal deadline. Administrators are advised to upgrade promptly or restrict public surveys and related routes until patches are applied.

0
ProgrammingDEV Community ·

METR's Updated Time Horizon Plot Shows AI Coding Gains Outpacing Judgment Tasks

METR, an AI evaluation nonprofit, has published an updated time horizon plot measuring how long AI models can autonomously complete tasks before failing. For software engineering benchmarks, the plot shows the time horizon doubling roughly every seven months since 2021, growing from seconds to over an hour. The plot only tracks tasks with clear right answers, such as code that compiles or fails. METR notes that models lag on judgment-based tasks, whose time horizons are shorter and improving more slowly. The findings suggest coding automation will advance rapidly, while judgment-heavy work still requires human oversight.

0
ProgrammingDEV Community ·

Three Browser-Side Layers That Make Copying JavaScript Tools Harder

A developer explains that code running in a browser cannot be fully hidden, since visitors download and execute it. Instead, they describe a three-part approach for protecting browser-based tools: a domain lock that redirects copied pages back to the original site, minification to make the code hard to read, and edge checks to block scrapers and bandwidth thieves. The author acknowledges that none of these measures is unbreakable, but argues they remove the quick payoff most casual copiers seek. The setup is applied in a build step on a static Next.js site hosted on Vercel, where readable source stays private and only processed files are deployed. The article also shares practical lessons, such as avoiding renaming top-level functions that HTML event handlers depend on and skipping pre-minified libraries.