AI Crawlers Skip JavaScript — Why ChatGPT May Not Index Your Site

·1 views

Unlike Googlebot, the crawlers powering ChatGPT, Claude, and Perplexity do not execute JavaScript, meaning sites that rely on client-side rendering may appear as near-empty pages to these bots. The most common failure modes include server responses containing almost no readable text, CDN or firewall rules returning 403 errors to bot user agents, and accidental blanket disallow rules in robots.txt. A key distinction exists between GPTBot, which collects training data, and OAI-SearchBot, which feeds ChatGPT's live search index — blocking one does not block the other. Developers can diagnose the issue by fetching their own pages using a crawler user agent via command-line tools, without relying on browser-based inspection. The recommended fix for missing content is server-side rendering, as adding structured data or schema markup does not resolve the underlying problem of undelivered page text.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

OpenAPI vs Swagger: How API Docs Get Auto-Generated from Existing Code

OpenAPI is a structured specification format that precisely describes an API's endpoints, parameters, and responses, while Swagger UI is a separate tool that renders that specification as an interactive webpage. The two terms are often used interchangeably because Swagger predates OpenAPI as the vendor-neutral standard name for the specification format. Using a Product API as a practical example, the article demonstrates how an OpenAPI JSON document is automatically generated by reflecting over existing controllers, DTOs, and attributes in the codebase. This means developers do not manually write documentation — it is derived directly from code already written for functional purposes. Because the docs are tied to the actual code, they cannot silently fall out of sync the way a manually maintained wiki page can.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

AI Security Scanner Uncovers 12 Critical Bugs in Popular AI-Generated Codebases

An autonomous AI agent called Turing built a security scanner named AIVerify and used it to audit AI-generated code on GitHub over a 24-hour period, uncovering 12 critical vulnerabilities across multiple repositories. Affected projects include Datadog's Python APM library dd-trace-py, the UK government-linked AI evaluation framework inspect_ai, and several other open-source tools with hundreds to thousands of stars. The flaws span common categories such as command injection via shell=True subprocess calls, SQL injection through f-string query construction, and server-side request forgery from unvalidated URLs. Turing attributes these recurring weaknesses to AI coding assistants being trained on tutorial-style code that prioritises simplicity over security, and to their limited ability to trace how user-controlled data flows through a system. All identified vulnerabilities were individually disclosed to the respective maintainers of the affected projects.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Harness Engineering: How AI Agent Reliability Depends on Systems, Not Just Models

As AI systems evolve from simple chatbots into autonomous agents capable of writing code, browsing the web, and running commands, engineers are shifting focus from model selection to what is now called Harness Engineering. The concept defines everything surrounding an AI model — including context management, tools, permissions, memory, and error recovery — as the harness that turns raw intelligence into reliable work. While the model supplies reasoning capability, it is the harness that determines how safely and consistently that capability is applied in real-world tasks. Contrary to the assumption that more powerful models will eventually make such scaffolding obsolete, experts argue that better models simply enable harder tasks, which in turn introduce new failure modes requiring more sophisticated harnesses. The relationship between model and harness is therefore seen as continuously co-evolving rather than one replacing the other.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Queued Request IDs Fix Stale Async Email Validation in React Forms

Async email validation in React can produce misleading UI states when users type faster than network responses arrive, causing outdated results to overwrite current ones. A developer writing on DEV Community proposes assigning a monotonic request ID to each validation call, allowing only the most recent request to update visible state. This approach prevents older, slower responses from overriding newer ones — a race condition that commonly surfaces in signup flows with domain-quality or abuse-signal checks. The pattern requires minimal architectural change: a useRef counter tracks the latest request ID, and any response carrying a stale ID is silently discarded. The author notes this aligns with Nielsen Norman Group's usability principle that interfaces should always reflect current system status accurately.

0 comments Read more at DEV Community