SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

97% of llms.txt Files Got Zero AI Bot Requests in May 2026, Ahrefs Data Shows

0
·1 views

A study by Ahrefs covering 137,210 domains found that 28% published an llms.txt file, but 97% of those valid files received zero requests during May 2026. Of the small fraction that were fetched, roughly 19.5% of requests came from named AI tools, led by GPTBot and Claude-Code — both coding assistants rather than search or retrieval crawlers. Google has since clarified in its AI optimisation guidance that these machine-readable files have no effect on Search rankings. The llms.txt specification was also quietly revised on 10 August 2026 without a public announcement, a change many commentators appear to have missed. Separately, a validator bug highlighted in the same analysis revealed that poorly written checker tools can certify malformed llms.txt files as valid, underscoring reliability concerns around the emerging standard.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

AI Features Carry a Hidden Energy Cost Developers Should Design Around

Every AI operation — from text prompts to image generation — consumes real physical resources inside data centers, including processors, cooling systems, and networking equipment. While a single request appears trivial, the energy footprint grows significantly at the scale of millions of calls. Developers have direct control over how frequently their apps query AI models, how much data is sent per request, and whether a lighter-weight tool or cached result could serve the same purpose. Thoughtful design choices — such as batching background jobs, setting data retention limits, and avoiding redundant requests — can meaningfully reduce resource consumption. As AI becomes standard in software, energy efficiency is emerging as a legitimate measure of engineering quality alongside speed, accuracy, and cost.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

How Structured Tracker Tasks Revealed Hidden Costs in AI-Assisted Development

Software engineer Anton, working on migrating a PHP monolith to Go services, observed that individual AI executor iterations were consuming around 350,000 tokens — far exceeding initial estimates. Investigation revealed the excess was driven by redundant context-gathering, where each executor independently rediscovered the same file paths, type names, and conventions without a shared record. To address this, Anton updated his specification format to include a closed list of files per iteration and a dedicated 'facts of the set' section, limiting file reads to at most three named files and prohibiting open tree searches. The core argument is that a tracker task must carry both a pre-work estimate and a post-work recorded fact, otherwise cost overruns remain invisible and project discussions devolve into subjective impressions. Anton notes the problem only became diagnosable because the task structure made the discrepancy measurable — turning what would have been a silent inefficiency into a documented spec defect with an attached cost.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

How One Team Built a Security Baseline Using Its Existing Elastic Stack in 3 Weeks

A software product company with 70–100 developers had cloud and Kubernetes infrastructure but lacked meaningful security visibility despite already running an Elastic/ELK deployment. The team had no consistent monitoring of failed authentications, RBAC changes, or suspicious container activity, and no defined process for responding to security incidents. Rather than adopting an expensive commercial SIEM, a consultant spent roughly three weeks building a right-sized security telemetry baseline using the existing stack. The project delivered prioritized detections, practical dashboards, and a lightweight incident-response workflow the DevOps team could own. The core takeaway was that security maturity can begin by making better use of systems already in operation, not by adding new tooling.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

How 178 simultaneous trip report uploads stress-tested a solo developer's LLM pipeline

A traveller imported 178 trip reports at once from a two-year Canada-to-Chile journey, overwhelming a small travel site that previously held only 9 reports. Each published report triggers three background jobs — text moderation, place extraction, and image moderation — all of which call an AI model via a queue. The system's worker processes ten jobs per minute to stay within the model's rate limit, meaning the full burst took roughly 35 minutes to clear, delaying any other users publishing during that window. The next morning, around 98 photos were flagged for human review because the image moderation model had repeatedly failed to respond, exhausting all three retry attempts. The developer traced each failure through stored status reasons and run logs, distinguishing genuine errors from expected system behaviour such as cron timeouts that only appeared problematic on the surface.

0 comments Read more at DEV Community
97% of llms.txt Files Got Zero AI Bot Requests in May 2026, Ahrefs Data Shows · ShortSingh