SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Can AI crawlers read your site? I checked mine

0
·6 views

First published on yappatyih.com. An AI crawler access check is a test of whether the bots behind ChatGPT, Claude, Perplexity and the others can fetch your pages, by requesting a page with each bot's name and reading the status code. On 30 September 2026 I ran one on this site: eight crawler names, eight answers of 200. Nothing was keeping the machines out. What the same audit found instead was a line in my own robots.txt, Disallow: /og/, that blocked the preview image of every page without a cover, plus a sitemap address that returned 404, one company with two ids, and dates that could claim

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

From JavaScript Basics to Deployed Websites: A Structured Learning Path for Aspiring Web Developers

Technical Reconstruction: From JavaScript Basics to Deployed Websites Transitioning from basic JavaScript knowledge to building and deploying a functional website is a complex journey that demands a structured learning path. Without a clear roadmap, learners risk wasting time on irrelevant resources, missing critical skills, or becoming overwhelmed by the complexity of web development, potentially abandoning their goals. This section dissects the essential steps, tools, and concepts required to bridge this gap, emphasizing causality and practical implications. Impact: Establishes core programm

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Microsoft SharePoint exposure: 163,266 matching assets and a code injection flaw that needs only a user account

Microsoft SharePoint exposure: 163,266 matching assets and a code injection flaw that needs only a user account SharePoint is one of the few enterprise platforms where an ordinary authenticated user is a meaningful attack position. The exposure figure for SharePoint therefore needs to be read alongside its authentication model rather than in isolation. Every count in this article comes from ZoomEye international, queried through the official Python SDK on 3 October 2026 between 02:33 and 02:35 UTC. All queries used the sub_type=all scope, which covers devices and websites, with a page size of

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

A Valid PNG Can Still Be the Wrong Tesla Wrap: Debug the UV Map First

Your file is PNG, its dimensions look right, and the export finishes without an error. Then a stripe lands on the wrong body panel. A valid image can still use the wrong template or place artwork in the wrong mapped region. A Tesla custom wrap is artwork for a digital vehicle visualization. Treat its template as a mapping contract: select it before drawing, preserve its canvas, and inspect where the pixels land before diagnosing the transfer process.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Colspan tables shredded my HTML-to-Markdown output — Markdown can't express merged cells

A user fed a cloud vendor's pricing page through my HTML-to-Markdown endpoint and got back a table where every plan's price sat in the wrong column. The page's comparison table used colspan for a tier header spanning three cells and rowspan for a plan name covering two rows. My converter processed each row independently — row count became pipe count. Rows under a colspan came out shorter, and whichever Markdown parser consumed the file aligned what came next with whatever column was open. A RAG pipeline downstream then quoted the wrong plan for a feature, which is how I found out: the answer l

0 comments Read more at DEV Community