How your robots.txt file silently blocks AI tools from ever citing your site

·6 views

A website's robots.txt file determines whether AI platforms like ChatGPT, Claude, and Perplexity can cite it in answers, yet most sites configure this file without fully understanding the consequences. There is a key distinction between training crawlers, such as GPTBot and ClaudeBot, and retrieval agents like OAI-SearchBot and PerplexityBot — blocking the latter permanently removes a site from AI-generated citations. A 2024–2025 wave of blanket AI bot blocking led many publishers to inadvertently restrict both types, with audits finding major outlets including the New York Times, Reuters, and Wired allowing fewer than three of seven retrieval agents. CDN providers like Cloudflare can also silently inject rules into the served robots.txt file that contradict or override a site owner's own settings, sometimes even adding legal declarations the owner never authored. Experts recommend running a simple curl check on your live robots.txt to identify unintended blocks and ensure retrieval agents are explicitly permitted if AI citation visibility is a goal.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

AI Agent Framework Choice Has Negligible Impact on Security, Study Finds

A controlled study involving 7,020 payload-verified trials across 6 large language models and 6 agent execution conditions found that the choice of AI agent framework — such as LangChain or CrewAI — accounts for only 0.06% of variance in security outcomes, a statistically insignificant result. In contrast, the type of attack used explained roughly 29% of variance, while the underlying model accounted for about 4%. The research was conducted to test whether framework selection meaningfully affects security in agentic AI systems, a common assumption among developers. The study is part of a series of four connected preprints also examining the reliability of automated detectors used in AI red-teaming pipelines, which the author found to be less dependable than widely assumed. All preprints are published under a CC BY 4.0 open license, with evaluation code available as open-source.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Security checklist for importing external AI agent definitions safely

Developers using AI coding tools like Claude Code or Codex often import external agent definition files from public collections without fully vetting them, which poses security and compatibility risks. Security-focused guidance recommends treating every external agent file as untrusted input, reading it in full rather than relying on its README before deployment. Reviewers should verify what the file claims to read, write, and execute against what the host environment actually permits, and stage files in a read-only location before placing them in any auto-watched directory. Changes should be minimal and categorised as either mechanical formatting fixes or deliberate meaning-altering decisions, each requiring a stated reason and an owner. Adopting an agent should also be contingent on clearly defining the specific gap it fills — compressed to a single sentence covering input, expected output, and explicit restrictions — before any file is added to a live agents directory.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer builds CSS-only poutine art with 50 fries, curds, and animated gravy

A developer created a detailed CSS art piece of poutine for DEV Community's Frontend Challenge: Comfort Food Edition, using no images, SVG, or canvas. The artwork features 50 fries, 13 cheese curds, and 11 gravy patches, with every element rendered purely through HTML and CSS. Each fry uses custom CSS variables to control its position, rotation, size, and individual browning level via color-mix() in the oklab color space. The only JavaScript present is roughly 20 lines powering a single 'Add more gravy' button that replays an animation and adds a cheese curd. The developer also shared key technical lessons, including why vmin is unreliable for responsive CSS art and why defining a scene-relative unit produces more consistent results across screen sizes.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Builds Accessible Poutine Landing Page Using Pure CSS and No Frameworks

A developer created 'Gravy Boat,' a fictional poutinerie landing page submitted to DEV Community's Frontend Challenge: Comfort Food Edition. The interactive hero features a fully CSS-built poutine — with 50 fries and 13 curds rendered without images — controlled by a slider that adjusts gravy levels from dry to drenched. The project uses just three files with no build tools, dependencies, or JavaScript frameworks, relying on CSS custom properties and a single data attribute to drive all visual changes. Accessibility was a core focus, with all color contrast ratios verified against actual rendered values, screen reader announcements added for filters and the gravy slider, and keyboard navigation fully supported including focus management. The developer also tested the page with JavaScript disabled, discovering and fixing gaps to ensure progressive enhancement worked as intended.

0 comments Read more at DEV Community