38 of Top 50 News Sites Are Invisible to AI Citation Bots, Audit Finds

·1 views

An automated audit conducted on August 1, 2026 tested the 50 largest English-language news websites to determine whether AI retrieval agents — the bots that enable tools like ChatGPT and Perplexity to cite and link sources — could access their content. The analysis evaluated 15 AI crawlers across each site's robots.txt file and checked whether homepages served readable text without JavaScript execution, since retrieval agents do not run JavaScript. Only 12 of the 50 sites passed both conditions and were deemed fully citable. A striking pattern emerged: 25 sites block retrieval agents while permitting at least one training crawler, meaning their content may be used to train AI models but will never be cited or linked in AI responses. Three sites — NPR, Politico, and The Information — allow all crawlers but render no readable text without JavaScript, making them uncitable despite open access policies.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Hard-Learned Celery Production Tips: Monitoring, Retries, and Silent Failure Fixes

A software team sharing lessons from four years of Celery deployments describes how a silent worker failure once left 40,000 tasks unprocessed and delayed client invoices for two weeks. The core issue is that Celery, by default, acknowledges tasks before they run, meaning a worker crash silently drops the job with no alert raised. Setting task_acks_late and task_reject_on_worker_lost ensures failed tasks are requeued rather than lost, while worker memory and task-count limits prevent gradual degradation. The team also recommends deploying Flower with event tracking enabled to monitor queue depth and active worker counts, triggering alerts when thresholds are breached. For task reliability, they use exponential backoff retries with differentiated handling for expected errors versus genuine data-missing cases.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

REST vs GraphQL: How to Choose the Right API Style for Your Project

REST and GraphQL are two widely used API approaches, each with distinct trade-offs depending on the use case. REST organises data around fixed endpoints and HTTP methods, making it simple to implement, cache, and version, but it can lead to over-fetching or multiple round-trips for related data. GraphQL lets clients request exactly the fields they need in a single query, which benefits mobile apps on variable networks, though it introduces added complexity in setup, caching, and query management. Many teams adopt a hybrid approach, using REST for public-facing APIs and GraphQL internally to reduce frontend round-trips. The best choice depends on team familiarity, client needs, and whether the API is intended for public or controlled consumption.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How 'Codeburn' and Language Server Protocol Aim to Fix AI Coding Hallucinations

AI coding agents like Devin, Cursor, and Copilot Workspace frequently generate syntactically incorrect or logically flawed code — a problem known as hallucination — because they predict tokens based on training data rather than a repository's actual structure. Each hallucination wastes computational resources and pollutes the agent's context window, causing cascading errors in subsequent reasoning steps. The emerging 'Codeburn' architectural pattern attempts to address this by discarding invalid code paths in real time, before they are committed, using feedback from the development environment. Central to this approach is the Language Server Protocol (LSP), which supplies live data on abstract syntax trees, types, and dependencies to constrain what the model can generate. Together, Codeburn and LSP aim to shift the burden of code correctness away from probabilistic language models and onto deterministic static analysis tools.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How OSPF Hello Packets Filter Neighbor Relationships Before Adjacency Forms

An OSPF Hello packet does more than act as a keepalive sent every 10 seconds — it functions as a strict filter that determines whether two routers can begin a neighbor relationship at all. Before any adjacency or database exchange can occur, four fields in the received Hello packet must exactly match the local interface configuration: area ID, netmask, hello interval, and dead interval. If any of these fields differ, the relationship silently never starts, with no error sent to the remote router and often no visible log entry. This behavior explains a common troubleshooting scenario where two routers on the same segment both send Hellos yet never form an adjacency. A free educational series called 'Protocol in Code' explores this logic through a compact Python implementation available on GitHub.

0 comments Read more at DEV Community