Ethical Web Scraping: Why Respecting Robots.txt and Rate Limits Is Non-Negotiable

·1 views

Web scraping is legally permissible for publicly accessible data under a 2023 US court ruling in hiQ Labs v. LinkedIn, but only when site terms of service and authentication boundaries are respected. A 2024 Zyte survey found that 68% of scraping projects fail within the first month due to IP blocks, with a further 23% failing from legal threats. Experts warn that aggressive crawling behaviors — such as sending dozens of requests per second or spoofing browser identities — trigger firewalls, IP bans, and potential violations of the US Computer Fraud and Abuse Act or EU GDPR. Best practices include parsing robots.txt files before crawling, maintaining request delays of one to five seconds, and using transparent bot identifiers in User-Agent strings. A 2022 Cloudflare study found that crawlers observing a delay of at least one second encounter 90% fewer bot-blocking events than those making sub-second requests.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Seven Lessons From Migrating a Multi-Tenant SaaS Dashboard From MUI to Shadcn

An engineering team replaced Material UI with Radix and Shadcn across a multi-tenant SaaS dashboard over the past quarter, reducing bundle size and simplifying their design system. The migration involved six teams and dozens of production components, and proved far messier than typical online tutorials suggest. Key issues included focus management differences in dialogs, incorrect component mappings such as using Drawer instead of Sheet, and Radix portal rendering bugs caused by CSS transform properties on ancestor elements. The team also had to build a custom multi-select and async-select solution on top of Radix Popover and Command, since Radix Select lacks native async and multi-value support. Their main takeaway was that a real-world migration rarely achieves full coverage, and maintaining clear escape hatches for complex legacy components is more practical than forcing a complete rewrite.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How PostgreSQL Internally Processes a SELECT Query: From SQL to Rows

When a PostgreSQL query like SELECT * FROM users is submitted, the database runs it through several internal stages before returning results. First, a parser checks syntax and builds a parse tree, then an analyzer verifies that referenced tables and columns actually exist. A rewriter may transform the query, after which the planner evaluates possible execution strategies — such as a full table scan versus an index scan — and picks the least costly one. The executor then carries out the chosen plan, retrieving data via a buffer cache called shared_buffers before falling back to disk if needed. Understanding this pipeline helps developers use tools like EXPLAIN more effectively to diagnose and improve query performance.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer finds 96 routing policy violations in AI agent logs, all off the main thread

A software developer auditing their AI agent's token usage discovered that 96 out of 425 routing decisions — about 22.6% — deviated from their stated policy, costing an extra $1,248.13. All deviations occurred in subagent threads, not on the main thread, revealing a gap between intended and actual model selection behavior. The developer maintains a 30-line YAML routing policy and an append-only trace log, comparing expected versus actual model tiers per decision unit. An early policy draft was found to be misaligned with actual intent, prompting a correction before measurements were taken to avoid scoring behavior against a flawed baseline. Separately, trace logs revealed an 18-day gap in Claude Fable 5 usage that appeared to reflect natural session drainage rather than a deliberate choice to stop using the model.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

HTML's native <dialog> element replaces focus traps, ARIA hacks, and backdrop divs

A DEV Community article highlights how developers commonly build modals using custom divs loaded with ARIA attributes, focus-trap libraries, and manual keyboard listeners — work the browser can handle natively. The HTML <dialog> element, opened via showModal(), automatically manages focus trapping, Escape-to-close behavior, and ARIA roles without any third-party dependencies. It also renders a built-in backdrop styled via the ::backdrop pseudo-element, eliminating the need for manually positioned overlay divs. Developers can add entry and exit animations using @starting-style and CSS transitions, with a small JavaScript fallback for older browsers. The article concludes that adopting <dialog> reduces boilerplate significantly while improving accessibility out of the box.

0 comments Read more at DEV Community