SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

JavaScript-Only Links Block AI Crawlers From Discovering Web Pages, 41-Day Test Finds

0
·3 views

SEO engineer Vinicius Stanula ran a 41-day field experiment on a roughly 1,000-page test site to measure how different crawlers handle JavaScript-injected navigation links. Pages linked only through JavaScript were entirely missed by AI bots including GPTBot, ClaudeBot, and PerplexityBot, while Googlebot could follow such links but with declining coverage at greater depths. When all links were switched to plain HTML, GPTBot discovered 250 new pages within 48 hours and Bingbot found 89 more, confirming the earlier gap was caused by JavaScript rendering limitations. The findings highlight that content rendered for human visitors via client-side scripts can remain invisible to crawlers that do not execute JavaScript. Stanula shared the experiment on LinkedIn as a practical warning for organizations seeking consistent content discovery across both traditional search engines and AI-driven platforms.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

New Quality Standard Proposes Audit Checklist for LLM Memory Architecture

A versioned audit checklist titled 'Memory Architecture Quality Standard for LLM Assistants' (v1.1, dated August 21, 2026) has been published to help developers evaluate long-term memory systems in AI assistants and agents. The standard covers the full memory lifecycle — from input ingestion and storage to retrieval, context assembly, and feedback loops — across relational databases, vector indexes, caches, and graph stores. Items are categorized by criticality, with blockers flagged for risks such as data poisoning, memory leakage, secret exposure, and uncontrolled memory growth. The checklist requires auditors to map all storage systems and verify read/write ownership before proceeding through sections covering input validation, retrieval logic, and prompt injection risks. Each finding must be backed by concrete evidence such as file references, SQL query results, or prompt snapshots, as unsupported assertions are not considered verified.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Graph Database Benchmark Reveals Memory Limits and Surprising Performance Gaps

A developer benchmarking six graph databases under strict resource constraints — 0.5 vCPU and 256MB RAM — found that Memgraph and ArangoDB both crashed before completing the test. Memgraph failed due to its in-memory transactional engine requiring more memory than the 256MB cap allowed, while ArangoDB miscalculated its cache size by reading the host machine's RAM instead of the Docker container's limit. To keep the benchmark valid, the developer added Kùzu, an embedded graph database, which peaked at just 1.94MB of memory while handling the same dataset. Despite lacking traditional index support, Kùzu outperformed cloud-hosted AuraDB's indexed lookups by roughly 40 times, largely exposing internet latency as the dominant cost in cloud database queries. The findings highlight that no single graph database excels across all workloads, with FalkorDB winning on traversals but underperforming on aggregation tasks.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

A misplaced 'finally' clause silently broke an AI phone agent after every first reply

A developer building an AI receptionist for a real landline discovered that callers were greeted correctly but the agent stopped responding after the first exchange, with no errors or exceptions raised. The WebSocket connection to the speech provider remained healthy throughout, making the bug nearly invisible to standard monitoring. The root cause was a Python 'finally' block inside an async generator that marked the connection as closed whenever a 'break' statement exited the loop at the end of each conversational turn. Since the agent checked the 'is_connected' flag before listening for follow-up audio, it silently refused to process any further input despite the socket being fully operational. The fix involved removing the cleanup logic from the generator itself and delegating connection teardown to a dedicated close method, ensuring a consumer breaking out of iteration no longer mimicked a genuine disconnection.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Developer builds dual-LLM review system after 157 agent plans reveal planning flaws

A developer running 157 AI agent plans against a real large language model found that most failures originated in flawed planning, not flawed execution. To address this, they built PlannerCritic, a system that separates plan drafting and plan review across two distinct LLMs to avoid self-reinforcing errors. Deterministic structural checks run first, verifying ordering, rollback coverage, and preconditions without reading goal text, making them resistant to prompt injection. The system operates within a bounded revision loop, and if consensus cannot be reached, it escalates to a human with a single targeted question rather than guessing. A real-world test involving a blockchain chain-split recovery task demonstrated the approach, with the critic flagging multiple sequencing blockers in a plan that had initially appeared sound.

0 comments Read more at DEV Community