Most websites don't block AI crawlers, yet almost none appear in AI answers

·1 views

A new study from Website Auditor reveals a stark disconnect between AI crawler access and citation rates across the web. Only 8.9% of websites have taken steps to block AI crawlers from indexing their content. Despite this open access, 94.8% of those sites are never referenced or cited in AI-generated answers. The findings suggest that allowing AI bots to crawl a site does not translate into visibility within AI responses. The research highlights a growing concern for publishers about the value of their content being used but not credited by AI systems.

Read the full story at Hacker News

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Dev Tutorial: Implementing Paginated Property Listings with RTK Query

A developer tutorial published on DEV Community demonstrates how to build server-side pagination for a property listing web application using Redux Toolkit Query (RTK Query). The guide covers setting up a typed API slice with endpoints that accept page number, page size, search term, and status filter parameters. A React component called PropertyTable manages local state for pagination and search, using a debounce effect to limit API calls as users type. The tutorial also shows how to calculate total pages from the API response and conditionally render UI controls based on the logged-in user's role. The implementation targets a localhost REST API and uses Bearer token authentication via request headers.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why Documentation Should Be Treated as Part of Software Architecture

A software engineering perspective argues that documentation is not a post-development task but a core component of system architecture, on par with code itself. Undocumented design decisions — such as why a soft delete was chosen over a hard delete, or why a specific retry strategy was adopted — represent hidden constraints that future developers can unknowingly violate. Writing design documents before or during implementation, rather than after, forces engineers to surface flawed assumptions early, before costly code investment makes it harder to challenge an approach. Architecture Decision Records (ADRs) are highlighted as a practical tool for capturing the reasoning behind decisions, helping future engineers understand context that no compiler or code review can convey. The article concludes that treating documentation as optional leads to system decay even when all tests pass, because correctness of logic and correctness of intent are two entirely different things.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Four SQL Patterns That Power Social Network Features at Scale

Building social features like follows, feeds, and mutual connections may appear simple but requires well-known SQL query patterns to function reliably at scale. A follow relationship is typically stored in a basic adjacency table, but self-joins on that table become costly when high-follower accounts generate hundreds of thousands of intermediate rows. Feed timelines are handled via two strategies: fan-out on write pre-inserts posts into each follower's feed for fast reads, while fan-out on read computes the feed dynamically at query time, with production systems combining both approaches depending on follower count. Mutual-friend lookups are graph problems that strain standard SQL joins, and better-scaling solutions involve representing follow lists as sorted arrays or bitsets and intersecting them outside the join layer. Proper indexing in both directions, early use of LIMIT clauses, and hybrid fan-out strategies are the key engineering levers for keeping these queries performant.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Guide to Building QWeb PDF Reports in Odoo 19 Covers Beginner to Advanced Techniques

A comprehensive tutorial published on DEV Community walks developers through creating professional PDF reports in Odoo 19 using QWeb, the platform's XML-based templating engine. The guide explains the full report generation pipeline, from a user clicking 'Print' through ir.actions.report and QWeb template rendering to final PDF output via wkhtmltopdf. It covers foundational topics such as setting up module structure, registering report actions, and using core layout components like web.external_layout and div.page. Advanced techniques addressed include reusable templates, custom paper formats, barcodes, QR codes, multilingual and RTL support, and performance optimization. The tutorial aims to equip Odoo developers with best practices for building maintainable and scalable reports across use cases like invoices, purchase orders, and payroll documents.

0 comments Read more at DEV Community