SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Medium blocks major AI training crawlers but remains open to AI search indexing

0
·138 views

A developer discovered that Medium's robots.txt file explicitly blocks eight AI-related crawlers — including GPTBot, ClaudeBot, and Bytespider — preventing bulk data collection for model training. However, the platform remains accessible to search-grounded retrieval bots such as Googlebot, Bingbot, PerplexityBot, and OAI-SearchBot, meaning Medium articles can still appear as cited sources in AI-powered search tools. Notably, Google-Extended and CCBot are also left unblocked, leaving open indirect pathways to AI training through Google's existing crawl data and the Common Crawl archive. The finding prompted the developer to abandon plans for a Medium column on AI search visibility, since the platform's setup works against the strategy being advised. A comparison of similar platforms — including DEV, Substack, and Hacker News — shows none have singled out AI crawlers the way Medium has, while Reddit takes the opposite approach by blocking all crawlers and licensing its archive directly.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Nvidia GPU leasing deals are pushing up prices, and a Python tool can track the drops

Nvidia's GPU leasing and repurchase arrangements, run through banks and specialized funds, let customers rent cards and return them after a set term. Returned units then cycle back into the secondary market, keeping supply tight and lifting used prices that serve as benchmarks for new ones. An 8-K filing reportedly indicates that 30% of RTX 4090 and H100 units sold in the US in October 2026 were sold under such contracts. Reported price increases from October to November 2026 ranged from about 12% to 19% across new and used RTX 4090 and H100 cards. The article also offers a Python script that monitors prices and sends an alert when a GPU falls below a chosen threshold.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Wisp: A tiny, dependency-free Go static file server with Docker image of about 3 MB

Wisp, originally codenamed LWS (Lightweight Web Server), is a small static file server written in Go using only the standard library. Its author built it because existing web servers are either too large and complex or too limited for modern deployments. The project's Scratch container image is roughly 3 MB, starts almost instantly, and has no runtime dependencies. Wisp includes features such as confined file serving, hidden-file protection, precompressed content support, SPA fallback routing, built-in Prometheus metrics on a separate port, and Kubernetes health probes. Its source code is available on GitHub, with setup driven by a simple JSON configuration file.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Solo developer ships a VPN app across iOS, macOS, Android and Windows without writing a protocol

A mobile developer released the Privalune VPN on iPhone, iPad, Mac, Android and Windows working alone. Rather than building cryptography or packet handling, the apps act as remote controls that give each operating system's built-in IKEv2 client a server and login. On Apple platforms this relies on NEVPNManager and on-demand rules that connect automatically on untrusted Wi-Fi, while Android uses the VpnManager API, which requires Android 11 or later. The Windows version is built with Tauri and sets up per-user profiles through PowerShell and the RAS API, with a mapped set of error messages for common connection failures. The backend runs as small TypeScript functions on Supabase Edge Functions, serving a cached server list that falls back to the last good copy if upstream fails.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Why password reset links must be signed, single-use, and time-limited

A password-reset link stayed valid for eleven days after the user had already changed their password, because the service embedded an unsigned user ID and timestamp in the URL. Anyone who saw that link, through a shared screen, screenshot, or browser history, could reset the account. The article argues that such links act as bearer credentials and must be unforgeable, time-bounded, and spendable only once. It presents a versioned, HMAC-signed token design in Python to meet these requirements. It also covers attack vectors such as replay, key compromise, leakage through logs and referrer headers, and error-message oracles.

0 comments Read more at DEV Community