1 in 10 Websites Have a Broken llms.txt File, Including Reddit

·9 views

A developer audit published on DEV Community on September 15 found that roughly one in ten websites have a misconfigured or broken llms.txt file. The llms.txt standard is an emerging convention designed to help AI language models better understand and navigate website content. High-profile platforms, including Reddit, were found to have faulty implementations of the file. The findings highlight a growing gap between the rapid adoption of AI-oriented web standards and the quality of their real-world deployment.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Medium blocks major AI training crawlers but remains open to AI search indexing

A developer discovered that Medium's robots.txt file explicitly blocks eight AI-related crawlers — including GPTBot, ClaudeBot, and Bytespider — preventing bulk data collection for model training. However, the platform remains accessible to search-grounded retrieval bots such as Googlebot, Bingbot, PerplexityBot, and OAI-SearchBot, meaning Medium articles can still appear as cited sources in AI-powered search tools. Notably, Google-Extended and CCBot are also left unblocked, leaving open indirect pathways to AI training through Google's existing crawl data and the Common Crawl archive. The finding prompted the developer to abandon plans for a Medium column on AI search visibility, since the platform's setup works against the strategy being advised. A comparison of similar platforms — including DEV, Substack, and Hacker News — shows none have singled out AI crawlers the way Medium has, while Reddit takes the opposite approach by blocking all crawlers and licensing its archive directly.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Pre-Launch AI Checklist: What Teams Must Verify Before Production Deployment

Deploying an AI feature to production requires far more than a successful demo, as real traffic exposes issues like bad inputs, token spikes, and slow dependencies that test environments rarely replicate. Teams should verify reliability through fallback paths, human escalation routes, and clearly defined SLOs before exposing any system to live users. Latency must be measured end to end — covering model response time, retrieval, network overhead, and time to first token — since staging performance often understates real-world slowness. Cost controls, including token usage tracking, query caching, and model selection reviews, should be built in early rather than addressed after launch. Security measures such as prompt validation, data redaction, secrets management, and access logging are considered essential from day one, alongside post-launch monitoring to catch gaps that testing alone cannot surface.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Build a 'Find Stores Near Me' Feature Using JavaScript Geolocation

Developers can build a store locator feature using the browser's Geolocation API, which returns a user's latitude and longitude after requesting permission. The coordinates are then compared against stored location data using the Haversine formula, which calculates straight-line distance between two geographic points. Results are sorted by proximity to surface the nearest stores first. Since users may deny location access or use unsupported browsers, the implementation should include a fallback option such as ZIP code or address search. For production use at scale, straight-line distance calculations may need to be replaced with a routing service to provide accurate driving distances.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why Tool Descriptions Are the Real API Contract in MCP Servers

A developer building a Model Context Protocol (MCP) server discovered that prose descriptions — not code or schemas — function as the true API contract, determining whether an AI agent selects and uses a tool correctly. Across a small three-tool sample maintained for just one week, the same tool description drifted between two files, with minor but meaningful wording differences emerging within days. The author warns that at scale — fifty tools, multiple teams, a year of changes — such description drift could cause an MCP gateway to describe backend behavior that no longer matches its actual implementation. Unlike REST APIs, which rely on routes, status codes, and diffable OpenAPI documents as a ground-truth contract, MCP servers have no such fallback, making accurate prose the sole interface. The piece concludes that the highest-value content in any tool description is provenance — explaining where parameter values must come from — since this is information a JSON schema alone cannot convey and its absence can cause agents to hallucinate identifiers.

0 comments Read more at DEV Community