Study: AI Crawlers Miss All JS-Only Pages, Find 100% of HTML-Linked Content

·1 views

A 41-day controlled experiment by Vinicius Stanula, reported by Search Engine Land, tested how AI and search crawlers handle JavaScript-injected versus standard HTML navigation links across a site of roughly 2,400 pages. GPTBot and ClaudeBot crawled all 748 HTML-linked pages but discovered zero pages accessible only through JavaScript navigation, while Bingbot and Meta External Agent also largely failed to penetrate the JavaScript group. The test isolated navigation as the sole discovery variable by disabling sitemaps, breadcrumbs, and other crawl paths. When JavaScript links were converted to HTML in Phase 2, GPTBot recovered around 250 previously missed pages within 48 hours, though Googlebot had rediscovered only one page by day 41. The findings highlight that relying solely on JavaScript for internal linking can effectively hide content from AI crawlers and many search bots, regardless of content quality.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

SEO Strategy Must Evolve as AI Answers and Zero-Click Searches Dominate Google

Search Engine Land's 2027 SEO roadmap warns that ranking first on Google no longer guarantees brand visibility, as AI-generated answers and zero-click results are reshaping how users find information. Data cited from Google I/O 2026 shows AI Overviews surpassing 2.5 billion monthly users, while 2026 studies estimate 68% of US Google searches end without a click. Marketers are advised to shift focus from simply winning search rankings to becoming credible, citable sources that AI systems can reference and recommend. The roadmap elevates the importance of brand mentions, digital PR and authoritative content alongside traditional signals like backlinks. Businesses are encouraged to maintain conventional SEO efforts while also optimizing for discovery across AI platforms, YouTube, Reddit and TikTok.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer builds open-source, offline time tracker using Python, Tkinter, and SQLite

A developer created Local Time Tracker, a free, open-source Windows desktop app that monitors application usage locally without requiring an account or sending any telemetry. The tool uses Python, Tkinter, and SQLite to record which foreground application is active, its window title, and periods of keyboard and mouse inactivity. All data is stored in a local SQLite database on the user's machine, avoiding the privacy risks associated with cloud-based time-tracking tools that may capture sensitive information from window titles. The app generates daily and seven-day activity summaries and can produce standalone offline HTML reports, with the developer noting that the database is not encrypted and relies on the security of the user's Windows account. The project was also prepared for external contributors, with the developer sharing architectural decisions and privacy trade-offs made during development.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Can AI Ever Truly Experience? The Philosophy of Machine Consciousness

The concept of 'qualia' — the subjective feel of experiences like pain, color, or emotion — raises a fundamental question about whether describing an experience is the same as having one. This philosophical puzzle is increasingly being applied to artificial intelligence, as AI systems grow more capable of simulating human-like responses. Philosopher Frank Jackson, who originally argued that subjective experience cannot be captured by scientific description, later reversed his position, deferring to science over intuition. The debate has shifted from a purely technical or philosophical one to an ethical question: what do we owe an entity that behaves as though it feels? Ultimately, the discussion reveals that humanity still lacks a complete understanding of consciousness itself.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Obsidian Copilot Plugin Lets Users Query Their Notes Using AI

A developer named Rijul has shared a method to turn an Obsidian note vault into an AI-powered knowledge assistant using the Copilot community plugin. The plugin can be installed directly from Obsidian's Community Plugins settings and connected to an AI provider of the user's choice. Once set up, users can ask natural-language questions about their past notes, such as recalling when or how they completed a specific task. The author demonstrated the workflow by querying his vault for details about a Microsoft Teams integration he had worked on previously. The approach aims to reduce time spent manually searching through notes by letting an AI agent surface relevant information on demand.

0 comments Read more at DEV Community