Developer Builds Polite Web Crawler With Rate Limiting Across Node Cluster Workers

·2 views

A developer built a small search engine from scratch, including a web crawler, an inverted index in MongoDB, and a BM25 ranker, to understand how such systems work internally. The initial crawler version lacked rate limiting and ignored robots.txt, meaning it would aggressively hammer web servers it encountered. To fix this, the developer implemented politeness controls, but discovered that rate limiting across Node.js cluster workers is a concurrency problem, not just a parsing one — and got it wrong twice before finding a correct solution. The key insight was that broken politeness features produce no errors on the crawler's side; only the targeted server experiences the impact as hostile behavior. The developer recommends testing by asserting what the remote server actually received, rather than relying solely on what the local code returns, and notes the full implementation is publicly available on GitHub.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How to Harden Your VPS With Firewall, SSH Config, and Fail2ban

Public servers face automated attacks within minutes of going online, including password brute-force attempts, port scans, and exploit probes. AhnLab's Q1 2025 report found that two malware families, P2PInfect and Tsunami, together made up around 80% of samples collected from compromised Linux SSH servers. A VPS typically sits behind two distinct firewalls: a provider-level network firewall managed via the hosting control panel, and a host-level firewall such as UFW or firewalld running inside the operating system itself. Security best practice recommends running both layers simultaneously, as each covers gaps the other cannot, rather than treating them as interchangeable. A default-deny firewall policy, hardened SSH settings, and Fail2ban together form a three-layer defence that can be configured in under an hour.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer builds AI-powered digital safety passport app for lost cats

A developer created Cat Guardian, a privacy-focused web app designed to help reunite lost cats with their owners, inspired by personally owning seven cats. Each cat receives a digital identity that includes health records, vaccination history, and an AI-generated visual profile built using Google's Gemini model. A QR code linked to a collar allows finders to access a public rescue profile without exposing the owner's private contact details. If the owner activates Lost Mode, a finder can submit a sighting, and the owner is notified via a secure backend relay. The open-source project was built with React, Supabase, and TypeScript, and supports both English and Brazilian Portuguese.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Four Common Swift 6 Concurrency Errors and How Developers Can Fix Them

Enabling Swift 6 strict concurrency mode on an existing project can surface dozens of errors, many of which represent genuine data races and actor isolation bugs that were silently present in production code. One of the most frequent mistakes is capturing non-Sendable reference types inside Task closures, which can be resolved by converting classes to structs or using actors for mutable shared state. Another common issue involves attempting to update MainActor-bound properties from within a Task that does not automatically inherit that isolation context, which requires explicit @MainActor annotations on the relevant class or function. Developers often suppress errors using nonisolated or @unchecked Sendable without understanding the underlying thread-safety implications, a practice that masks rather than resolves potential race conditions. A subtler pitfall is assuming that async helper functions called from a @MainActor class will automatically run on the main actor, when in fact isolation must be declared explicitly at the function level.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Scripted Triggers on SPI, I2C, and UART Cut Through Logic Analyzer Noise

Embedded hardware researchers often waste hours manually scrolling through millions of samples captured by logic analyzers when analyzing SPI, I2C, and UART buses. A scripted trigger approach lets analysts define precise conditions — such as a specific boot string or ASCII sequence — so the analyzer captures only the relevant window of data. For example, triggering on a UART string like 'U-Boot' can automatically record the two seconds that follow, which is how one researcher caught a smart plug exposing a root shell for just 400 milliseconds after boot. The same method can detect hardcoded credentials printed over UART, baud-rate switches mid-boot, and anomalous I2C address transactions that would otherwise be buried in repetitive sensor chatter. All described techniques are designed to run on low-cost hardware, such as a $20 logic analyzer paired with Python scripting tools like Saleae Logic or PulseView.

0 comments Read more at DEV Community