When to use a headless browser for scraping — and when a simple fetch will do

·8 views

A developer explains the practical decision-making process behind choosing between a headless browser and a simple HTTP fetch when scraping web data. The key test involves comparing the raw HTML response from a server against the live DOM rendered in a browser — if data appears only after JavaScript runs, a browser becomes necessary. While scraping a UK book gift-card site called National Book Tokens, the developer found the shop location data was absent from the raw HTML but present in a hidden DOM element populated by client-side JavaScript. Rather than launching a full Chromium instance inside a Firebase Function, the developer connected to a remote headless browser via WebSocket using puppeteer-core, keeping the browser's role minimal. The broader lesson is that headless browsers should handle only JavaScript execution and DOM settlement, with all subsequent data extraction done through faster, more reliable string operations.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Researchers Detail EEPROM Hijacking Method Targeting 8051 Microcontrollers

A technical methodology has been outlined for intercepting and redirecting execution flow in 8051-architecture microcontrollers, which remain widely used in constrained embedded systems. The technique involves inserting a binary hook — a long jump instruction — at the start of an SDCC-compiled validation function to redirect execution to a custom assembly routine stored in unused memory, known as slack space. A key challenge addressed is byte-alignment: replacing a 4-byte prologue with a 3-byte LJMP instruction requires a padding NOP to prevent program counter corruption and system crashes. The injected assembly routine conditionally bypasses a critical logic or security check only when a specific trigger value is detected in memory, otherwise resuming normal program execution. The approach is noted for its minimal memory footprint, making it particularly relevant in environments where available storage is measured in just a few kilobytes.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Key Engineering Lessons from Building Reliable Payments Infrastructure

A payments infrastructure engineer has shared hard-won lessons from scaling financial systems, emphasizing that unlike typical software, payment failures have immediate and serious consequences for users. The most critical lesson is idempotency: systems must be designed to handle duplicate requests safely, since retries, double-taps, and repeated webhooks are routine occurrences rather than edge cases. The engineer warns that a naive check-then-write approach leaves race conditions open, and that only database-level unique constraints can reliably prevent duplicate charges. Additional lessons cover treating payment gateways as failure-prone vendors, proactive monitoring to detect outages before customers do, and the unglamorous but essential work of ledgers, reconciliation, and state machines. The piece concludes that each of these principles involves tradeoffs that often conflict with one another, requiring careful engineering judgment.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why yfinance Fails to Fetch Natural Gas Futures Data and How to Fix It

Developers using the yfinance Python library have reported recurring failures when downloading natural gas futures data via the NG=F ticker on Yahoo Finance. The issue stems from yfinance being an unofficial scraping tool, meaning any backend change by Yahoo can silently break data retrieval without warning. Futures symbols like NG=F are especially prone to these disruptions, with multiple known issues logged by the community over the years. Three short-term fixes include upgrading the library, adding retry logic with backoff, and switching between yf.download() and yf.Ticker().history() methods. For production pipelines requiring reliable data, the author recommends migrating to a dedicated natural gas price API rather than depending on an unsupported scraping workaround.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

DNS gateway hit 500 QPS wall due to missing Bloom filter and SQLite bottleneck

HydraDNS, an open-source DNS security gateway built in Go, was falsely documented as using a Bloom filter for blocklist lookups, when in reality every DNS query triggered a SQL COUNT against a 92,000-row SQLite table. The shared single database connection serialized both blocklist checks and query logging, capping throughput at around 500 queries per second regardless of available CPU headroom. The developer discovered the phantom Bloom filter never existed in code — a planning note had gradually become accepted fact across multiple documents. Replacing the SQL lookup with an in-memory atomic domain map boosted throughput nearly 19x, from ~500 to ~9,500 QPS. However, removing that first bottleneck exposed a second: per-query goroutines for logging overwhelmed the single database connection at high load, causing memory to spike toward 2GB and risking out-of-memory crashes on target hardware.

0 comments Read more at DEV Community