How a URL Page Importer Handles Bot Walls, Fallbacks, and Asset Uploads

·1 views

A developer built a URL import feature for a page editor that clones any public webpage into an editable canvas, with all assets re-hosted on the user's CDN. The tool uses a two-tier capture system: a plain HTTP fetch for simple pages and a headless browser API for JavaScript-rendered content. Rather than silently degrading on all errors, the implementation distinguishes between configuration gaps, transient network failures, and upstream HTTP errors, ensuring users receive meaningful feedback when something goes wrong. To avoid storing bot-challenge pages as content, the importer uses a heuristic detector combining title strings, CSS class markers, and document size thresholds to identify Cloudflare and similar interstitial screens. The tier that produced each result is surfaced in the UI, so any invisible degradation does not become a misfiled support ticket.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How a pub quiz app fixed WebSocket reconnects that left players stuck for ages

A live pub quiz platform serving around 100 simultaneous players over a single venue WiFi discovered that brief two-second connection drops were causing phones to show 'Reconnecting' for far longer than expected. The root cause was a stale closure bug: the WebSocket onclose handler was reading React state inside a setState updater, which delayed side effects like scheduling a reconnect until after reconciliation. The fix involved mirroring status into a React ref so the handler could read current state and act on it immediately, outside React's update cycle. Separately, the polling fallback interval was doubled from five to ten seconds after engineers calculated that 100 phones polling every five seconds generated roughly 1,200 requests per minute from one IP address, exceeding rate limits and knocking out the entire venue. The team accepted slightly stale data in degraded mode as a deliberate trade-off, reasoning that a lagging leaderboard is less harmful than a complete service outage.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Five silent failures when automating real Chrome via CDP and how to fix them

A developer running Chrome browser automation using raw Chrome DevTools Protocol (CDP) on a persistent user profile documented five failure modes that produce no exceptions or error messages. Key issues included Chrome silently blocking popups not tied to recognised user gestures, requiring the --disable-popup-blocking launch flag to resolve. OAuth flows caused further problems because some providers navigate the current tab rather than opening a popup, meaning code that only watches for new targets stalls indefinitely. On shared browsers with multiple tabs, automation risked attaching to the wrong OAuth target, requiring pre-click snapshots of target IDs and opener verification. Input events such as clicks and keystrokes were silently ignored when the target tab was in the background, fixed by explicitly calling Target.activateTarget and Page.bringToFront before each interaction.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Metasploit Framework: Capabilities, Setup, and Common Lab Misconceptions

Metasploit is a widely used penetration testing framework that can be installed on Linux systems via standard package managers and configured with a backend database for session tracking. Once set up, users can run network scans using tools like Nmap to identify potential targets within a controlled lab environment. The framework allows testers to search for and deploy specific exploits, such as those targeting vulnerable services like vsftpd, paired with payloads for remote access. A key practical consideration is that Metasploit's effectiveness is limited against fully patched systems, making regular software updates a critical defensive measure. The guide, attributed to Alex J. on DEV Community, emphasizes using the framework responsibly within authorized lab settings.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer builds free client-side image resizer tool for Etsy marketplace sellers

A developer has released a free, browser-based image resizing tool specifically designed for Etsy sellers, available at etsy-image-resizers.netlify.app. The tool automatically resizes and pads product photos to the required 2000×2000 pixel white-square format and compresses them under 1MB as JPG files. Unlike conventional online converters, the tool processes images entirely on the client side, meaning no photos are uploaded to any external server. The project is open source and hosted on GitHub, with the developer actively seeking user feedback on the crop editor interface. Support for additional platforms such as Poshmark and Vinted is being considered, pending verification of their conflicting published image specifications.

0 comments Read more at DEV Community