Medium blocks major AI training crawlers but remains open to AI search indexing

·9 views

A developer discovered that Medium's robots.txt file explicitly blocks eight AI-related crawlers — including GPTBot, ClaudeBot, and Bytespider — preventing bulk data collection for model training. However, the platform remains accessible to search-grounded retrieval bots such as Googlebot, Bingbot, PerplexityBot, and OAI-SearchBot, meaning Medium articles can still appear as cited sources in AI-powered search tools. Notably, Google-Extended and CCBot are also left unblocked, leaving open indirect pathways to AI training through Google's existing crawl data and the Common Crawl archive. The finding prompted the developer to abandon plans for a Medium column on AI search visibility, since the platform's setup works against the strategy being advised. A comparison of similar platforms — including DEV, Substack, and Hacker News — shows none have singled out AI crawlers the way Medium has, while Reddit takes the opposite approach by blocking all crawlers and licensing its archive directly.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Which tools drop the color profile? Pillow, ffmpeg and sips across JPG, PNG and WebP

An ICC color profile is a small block of data inside the image file. It tells software which color standard the RGB numbers are meant to be read in. The same numbers read as Display P3 come out more saturated than when read as sRGB, and Apple's developer documentation notes that iPhone cameras can capture P3 starting with the iPhone 7. When the profile is missing, software generally falls back to sRGB. Nothing about the pixels changes.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Cline: Open Source Coding Agent With 67K GitHub Stars Rivals Claude Code and Cursor

Cline is an open source autonomous coding agent, licensed under Apache 2.0, that can read entire project structures, plan changes, edit multiple files, and run terminal commands while keeping the developer in control at each step. Originally launched as a VS Code extension called Claude Dev, it has expanded into four products: a VS Code extension, a JetBrains plugin, a CLI tool, and a Node.js SDK for building custom agents. Unlike proprietary alternatives, Cline supports a wide range of AI model providers — including Anthropic, OpenAI, Google, and local models via Ollama — letting users bring their own API keys. It features a two-stage Plan and Act workflow, project-specific rule files via .clinerules, and support for Model Context Protocol servers to connect external tools and services. The project has accumulated over 67,000 GitHub stars and more than 7,000 forks, signaling strong adoption within the developer community.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Nearly 4,000 Proxmox Hypervisor Consoles Exposed Online Amid Auth Bypass Flaw

A ZoomEye scan conducted in September 2026 found 3,988 hosts responding on port 8006, the default port for the Proxmox VE web management interface. This exposure coincides with a Proxmox authentication bypass vulnerability disclosed in August 2026, affecting versions 7.x through 8.0.3. An attacker reaching an unpatched console gains root-level access without needing to exploit additional vulnerabilities, making exposed instances high-value targets for ransomware and data extortion. Security researchers note the scan reflects exposure only and does not confirm which hosts are running vulnerable versions or who owns them. Defenders are advised to block port 8006 from public internet access and enforce multi-factor authentication on root accounts as immediate mitigations.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Builds Event-Sourced Library App in Under an Hour Using Sekiban DCB and AI

A developer used the Sekiban DCB Decider template alongside Codex and GPT-4o to rapidly build a library management application supporting book registration, borrowing, and returning. The project was scaffolded using a .NET 10 dotnet template and used PostgreSQL for storage with a Blazor-based frontend. Business logic was handled through Decider classes containing Validate and Evolve methods, ensuring consistency rules such as preventing duplicate borrowing of the same book. The AI was guided to follow existing sample implementations for Student, ClassRoom, and Enrollment features rather than introducing new patterns or abstractions. The entire process — including implementation, integration testing, concurrency checks against a live database, and UI fixes — was completed in less than one hour.

0 comments Read more at DEV Community