Most SaaS firms leave AI search bots open on robots.txt, while news sites block them

·3 views

A developer audited the home pages of 40 well-known websites in November 2026 to see how their robots.txt files treat 12 AI crawlers, and 31 sites yielded usable results. The crawlers were split into AI search and assistant bots, which fetch pages to answer users, and training bots, which collect content for model training. Ten of the 31 sites block at least one search or assistant bot, and all six news sites in the sample do so. None of the ten SaaS and developer tool companies block those bots, and nine of them publish an llms.txt file, which no news site does. The author suggests the split may reflect business models, since publishers risk losing visits to AI answers while software firms may benefit from mentions. The audit also revealed a bug in the author's own tool, which had wrongly flagged Figma as blocking AI search bots because Python's urllib.robotparser mishandles the end-of-URL marker and some bot-name matches.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

SimpleSAMLphp Metadata Signing Requires Managed Certificate Lifecycles

SimpleSAMLphp can sign the metadata it publishes and verify signed metadata from other entities, which makes certificates central to federation trust. A valid signature only shows that a known key signed the document, so operators must still decide which keys to trust and how metadata is refreshed and certificates rotated. The article advises using separate keys per entity and environment, keeping private keys restricted, and replacing any sample credentials before production. For rollovers, it recommends publishing the new verification certificate alongside the old one, waiting for consumers to refresh, and then removing the old certificate, while checking how cached metadata behaves when refreshes fail. Testing should cover malformed signatures, expired certificates, and unknown issuers, with monitoring for expiry, refresh failures, and validation errors.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Fujitsu Unit 1Finity Launches AI-Native VRAN to Run AI on Phone Tower Gear

1Finity, a telecom equipment and software unit of Fujitsu, has announced its full-stack AI-Native VRAN system is now available. The platform is built on NVIDIA Aerial, which virtualizes and accelerates radio access network functions. It aims to let carriers run mobile service and local AI workloads, such as factory or campus AI, on the same tower or edge hardware. The company is pitching the approach as a way for operators to earn additional revenue from existing infrastructure. However, 1Finity did not name any customers or publish performance figures in its announcement.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Builds Role-Based React Task Dashboard Using Context API and localStorage

A developer built EMS, an admin and employee task management app, to practice React state management without a backend. Persistence is handled entirely by localStorage, with no Express server or database involved. A single AuthProvider manages login state, user role, and task data, and admin and employee dashboards are rendered as separate component trees. The author isolated storage behind a utility module so the backend could be swapped later without changing the dashboards, and noted the approach isn't suited to multi-user or multi-device use. They invited feedback on whether the Context setup would hold up if a third role were added.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

YouTube Shorts pipeline's first five outcome verdicts all came back unjudgeable

A developer's automated YouTube Shorts pipeline produced its first five outcome verdicts, and all five were labeled unjudgeable. Four videos had no prediction on record, while the fifth had a prediction that missed actual views by roughly tenfold but still lacked the baseline file needed for judgment. The developer had made predictions mandatory only for one channel from October 25, leaving the other channels without that requirement. The core problem was that the input the pipeline required differed from what the verdict actually needed, including a same-window baseline. Fixes include a prediction gate on that channel, a daily sweep that labels outcomes, and a planned manual baseline re-measurement.

0 comments Read more at DEV Community