Open-source audit reveals how 18 major sites secretly filter AI crawlers

·3 views

A developer built an open-source tool called geo-crawl-audit to test how major websites treat AI web crawlers such as GPTBot, ClaudeBot, and PerplexityBot compared to regular browsers. Testing 18 prominent sites on August 7, the audit found that firewall configurations often contradict stated robots.txt policies, effectively making crawler access a business decision rather than a technical one. The New York Times, currently in litigation with OpenAI, blocked nearly all AI crawlers, while The Guardian, which holds a content deal with OpenAI, selectively allowed OpenAI bots but blocked Anthropic and Perplexity crawlers. Reddit's robots.txt disallows all AI bots, yet several were served live content anyway, exposing the gap between declared policy and actual enforcement. The audit also highlighted that most major AI crawlers do not execute JavaScript, meaning heavily client-rendered sites like LinkedIn and Reddit expose near-empty pages to crawlers regardless of access permissions.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Seven Lessons From Migrating a Multi-Tenant SaaS Dashboard From MUI to Shadcn

An engineering team replaced Material UI with Radix and Shadcn across a multi-tenant SaaS dashboard over the past quarter, reducing bundle size and simplifying their design system. The migration involved six teams and dozens of production components, and proved far messier than typical online tutorials suggest. Key issues included focus management differences in dialogs, incorrect component mappings such as using Drawer instead of Sheet, and Radix portal rendering bugs caused by CSS transform properties on ancestor elements. The team also had to build a custom multi-select and async-select solution on top of Radix Popover and Command, since Radix Select lacks native async and multi-value support. Their main takeaway was that a real-world migration rarely achieves full coverage, and maintaining clear escape hatches for complex legacy components is more practical than forcing a complete rewrite.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How PostgreSQL Internally Processes a SELECT Query: From SQL to Rows

When a PostgreSQL query like SELECT * FROM users is submitted, the database runs it through several internal stages before returning results. First, a parser checks syntax and builds a parse tree, then an analyzer verifies that referenced tables and columns actually exist. A rewriter may transform the query, after which the planner evaluates possible execution strategies — such as a full table scan versus an index scan — and picks the least costly one. The executor then carries out the chosen plan, retrieving data via a buffer cache called shared_buffers before falling back to disk if needed. Understanding this pipeline helps developers use tools like EXPLAIN more effectively to diagnose and improve query performance.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer finds 96 routing policy violations in AI agent logs, all off the main thread

A software developer auditing their AI agent's token usage discovered that 96 out of 425 routing decisions — about 22.6% — deviated from their stated policy, costing an extra $1,248.13. All deviations occurred in subagent threads, not on the main thread, revealing a gap between intended and actual model selection behavior. The developer maintains a 30-line YAML routing policy and an append-only trace log, comparing expected versus actual model tiers per decision unit. An early policy draft was found to be misaligned with actual intent, prompting a correction before measurements were taken to avoid scoring behavior against a flawed baseline. Separately, trace logs revealed an 18-day gap in Claude Fable 5 usage that appeared to reflect natural session drainage rather than a deliberate choice to stop using the model.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Ethical Web Scraping: Why Respecting Robots.txt and Rate Limits Is Non-Negotiable

Web scraping is legally permissible for publicly accessible data under a 2023 US court ruling in hiQ Labs v. LinkedIn, but only when site terms of service and authentication boundaries are respected. A 2024 Zyte survey found that 68% of scraping projects fail within the first month due to IP blocks, with a further 23% failing from legal threats. Experts warn that aggressive crawling behaviors — such as sending dozens of requests per second or spoofing browser identities — trigger firewalls, IP bans, and potential violations of the US Computer Fraud and Abuse Act or EU GDPR. Best practices include parsing robots.txt files before crawling, maintaining request delays of one to five seconds, and using transparent bot identifiers in User-Agent strings. A 2022 Cloudflare study found that crawlers observing a delay of at least one second encounter 90% fewer bot-blocking events than those making sub-second requests.

0 comments Read more at DEV Community