SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Websites May Be Invisible to AI Chatbots Due to Crawler Blocking and Rendering Issues

0
·5 views

A developer found that a site ranking well on Google was never mentioned by ChatGPT, Claude, or Perplexity when asked about its topic. Investigation revealed that the robots.txt file was blocking AI crawlers, the content relied on JavaScript rendering, and no structured data was present. The author built free online checkers to test robots.txt rules against 21 AI bots, show what crawlers see on a page, and score AI visibility. The article advises that site owners can explicitly allow AI bots like GPTBot, ClaudeBot, and PerplexityBot in robots.txt if they want their content used by AI tools.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Multi-Tenant Help Centers Need Privacy Controls That Outlast Search Vendor Swaps

A help-center answer must never cite material from another tenant, even when the underlying search backend is being replaced. The article recommends a staged retrieval design with explicit collections, bounded queries, and traceable source context, with tenant rules kept in the application contract so authorization logic survives a vendor change. Each retrieval unit would carry tenant, document, version, and access metadata, with the tenant filter drawn from the authenticated request rather than the user's question. Ingestion, querying, and citation are treated as separate observable stages, and missing citations or tenant mismatches are flagged as policy violations to be paged on. The piece also promotes Infrai as a possible adapter candidate offering a single REST API and key, though it presents this as one option among several.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Tutorial Shows How to Batch-Scan Class Essays for AI Writing with Python

A new DEV Community tutorial walks through building a Python script that scans a folder of student submissions for AI-generated writing. The script sends each file to the Proofademic AI Detector API, manages rate limits and asynchronous jobs, and caches results. It outputs a CSV file and a PDF report with a ranked class summary and sentence-level highlights. The author says a test run on eight sample submissions correctly flagged the ChatGPT-generated essays and found an AI-written paragraph hidden inside a human essay. The article also covers API setup, including free trial credits and pricing for the detector.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Spreadsheets Silently Accept Wrong Values, So AI Agent Edits Need Verification

Writing into an Excel or Sheets cell can appear successful even when the result is wrong, since cell addresses shift when rows are inserted and formulas can be overwritten with plain values. Cells also coerce data silently through formats, locale settings, or dropped leading zeros, while the write call still reports success. The author proposes that any edit be describable as a named target, a diffable change, reversible via a saved copy and hash, and checked by reading back recalculated values. A five-field record of each unattended write is recommended, including the range actually touched and a read-back of the computed result.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Why Valorant Built on Unreal Engine: Rented Tech Freed Riot to Focus on Competitive Integrity

Riot Games developed Valorant, a free-to-play 5v5 tactical shooter, on Unreal Engine 4 rather than a custom engine, a choice that let its team concentrate on servers and anti-cheat. The game launched on Windows on June 2, 2020, followed by PlayStation 5 and Xbox Series X/S versions in 2024. Riot later moved the live game to Unreal Engine 5.3 in July 2025 without changing its core gameplay feel. The article argues that using an existing engine let Riot prioritize 128-tick netcode and competitive fairness, which it describes as the game's central design goal. Valorant's approach is presented as a lesson for smaller developers weighing engine choices against building their own technology.

0 comments Read more at DEV Community