1 in 10 AI tools blocks GPTBot, but many owners may not know it

·6 views

A study of 9,037 live AI tools conducted on September 6–7, 2026, found that 10.5% explicitly block OpenAI's GPTBot in their robots.txt files, often without the site owner's direct intent. Hosting providers appear to play a significant role: 22.4% of Cloudflare-hosted sites blocked GPTBot, compared to just 5.2% on Vercel, suggesting platform-level toggles may be writing crawler policy automatically. Notably, 88.8% of sites that block GPTBot still allow OAI-SearchBot, a deliberate split that opts out of AI training while remaining eligible for ChatGPT search citations. Similar patterns were observed for Anthropic's ClaudeBot and Google-Extended, both blocked by roughly 10% of sampled sites. Researchers urge developers to manually verify their robots.txt after deployments or hosting changes, as the file functions as a content policy, not merely a technical configuration.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Proxmox Automation Series: Storage Validation, fstab Persistence, and Fifth Disk Workflow

A new module in the Proxmox Automation Series addresses three architectural gaps left open after Phase 2, which had configured LVM and XFS storage layers using Ansible and Terraform. The update moves disk layout definitions into a single source of truth file, group_vars/provisioned.yml, eliminating copy-paste drift risk between playbooks. A newly written verification playbook, 03_storage_verify.yml, confirms post-reboot filesystem persistence using validation tokens, refreshed facts, and assertions. Optional fifth-disk support was added via a dynamic Terraform block, allowing teams to test end-to-end workflows for adding new disks to a running environment. Scope was deliberately kept narrow, excluding dynamic inventory from Terraform state and multi-host scenarios, which are planned for later phases.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Tests Local AI Models Qwen and ModernBERT as Low-Cost Alternatives to OpenAI

A developer running a self-hosted RSS reader called RSSMonster began exploring local small language models after OpenAI API costs climbed to around €10 per month. To reduce expenses, they integrated two locally run models — Qwen, an Alibaba-developed embedding and language model, and ModernBERT, a classification-focused encoder — to handle tasks like summarization, tagging, and content quality scoring. Both models passed the project's existing semantic regression tests, though they produced noticeably different clustering and topic-assignment results compared to OpenAI's outputs. Qwen assigned more articles to events and topics than OpenAI, while ModernBERT handled classification tasks such as detecting promotional content and emotional tone. The experiment suggests that small language models running on common hardware can serve as viable, cost-effective alternatives for semantic features in personal or self-hosted projects.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Best of AI Launches Skills Directory to Centralize LLM Agent Tools

Best of AI has released a new Skills section on its platform, designed to help developers find and use packaged instruction sets for LLM agents like Claude. The directory addresses the longstanding problem of agent skills being scattered across repositories, gists, and forums by consolidating them in one browsable location. At launch, 50 curated skills are available across nine categories, including Code Review, Testing and Debugging, Document Automation, and Browser and Agent Automation. Skills are organized by category and author to help users quickly identify tools relevant to their workflow. The project is open source, and contributors are encouraged to submit new skills via a pull request or issue on the platform's GitHub repository.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer tests same static page on 8 free hosts, reveals key HTML handling differences

A developer deployed an identical 4.6 KB static page to eight free hosting platforms in a single day to document how each one handles uploaded HTML. All eight hosts — including Cloudflare Pages, GitHub Pages, Netlify, Surge, Hugging Face Spaces, and Lovable — preserved the HTML without modification, though several had notable caveats. Netlify now makes projects private by default, returning a 401 error with no clear warning in deploy logs, while GitHub Pages supports only one site per username rather than one per repository. Cloudflare Pages returns HTTP 200 with the index page for missing paths instead of a proper 404, and Surge.sh is not on the Public Suffix List, meaning all its subdomains are treated as a single site. The developer also flagged that programmatic page checks can be misled by bot-blocking responses from sites like npmjs.com and PyPI, recommending the use of a browser user-agent and anchor-count checks to verify deployments accurately.

0 comments Read more at DEV Community