Why Your Web Scraper Gets 403 on Servers but Works Fine Locally

·1 views

Web scrapers that run without issue on a local machine often return 403 errors when deployed to cloud servers or CI runners, even with identical code. Anti-bot systems evaluate requests across multiple signals simultaneously — including IP reputation, TLS fingerprint, HTTP/2 settings, and header order — before any HTML is served. Datacenter IPs from providers like AWS, GCP, and DigitalOcean are pre-flagged by most anti-bot vendors, while residential IPs are not, which explains the local-versus-server discrepancy. A silent 403 with no CAPTCHA or challenge page typically indicates a block at the network layer, meaning fixes like changing the User-Agent or slowing request rates will not help. The recommended fix is to switch to residential or ISP proxies first, then match a real browser's TLS fingerprint, and only introduce a full browser automation setup for pages that genuinely require JavaScript execution.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

A Lean Customer Success Playbook Built for Solo SaaS Founders

A former customer success manager at a 200-person SaaS company has shared a simplified retention framework tailored for solo founders managing between 30 and 150 customers. The system replaces enterprise tools like Gainsight with a five-signal health score that can be reviewed in under 15 minutes per week, covering feature activation, usage breadth, support sentiment, champion engagement, and billing health. Accounts are categorized into healthy, at-risk, or critical bands, each triggering specific outreach actions within defined timeframes. The playbook also outlines a structured rescue sequence for critical accounts, escalating from personalized emails to Loom videos and phone calls over 14 days. The author reports testing the approach across seven solo-founded SaaS products, noting that customers who adopted a second key feature within 45 days retained at more than twice the rate of those who did not.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Integration Tests Exposed 3 Unfixed Bugs That 90% Unit Test Coverage Missed

A developer building an MCP failure library — a shared memory tool that lets AI agents warn each other about recurring errors — discovered that three previously "fixed" bugs were still present after writing a real end-to-end integration test. Despite achieving over 90% unit test coverage across the SQLite layer, MCP transport, and retry logic, production failures occurred almost every other day due to untested component interactions. The integration harness spun up the actual server, used the real STDIO transport and SQLite database, and ran full scenarios without mocks, revealing issues including a cache queried before it finished loading, JSON-RPC payloads being split mid-stream, and a flawed deduplication query that returned random results after 150 database entries. None of these bugs were detectable in isolation because unit tests confirmed individual functions worked correctly but could not verify system-wide behavior. The developer also noted practical pitfalls such as accidentally locking the live database during testing, recommending the use of temporary files, process timeouts, and proper cleanup routines.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

useUpdateEffect Hook Lets React Developers Skip Redundant Mount-Time Effect Runs

React's built-in useEffect hook fires both on component mount and on subsequent updates, which can cause unintended side effects like premature toasts or analytics events. The useUpdateEffect hook from @reactuses/core solves this by wrapping useEffect and skipping the very first invocation, while keeping an identical API signature and cleanup semantics. Under the hood, it relies on a useFirstMountState primitive that flips a ref during render to track whether the component is mounting for the first time. Developers can drop it in as a direct replacement wherever mount-time execution is unwanted, with no new patterns to learn. One notable caveat is that in React 18 StrictMode during development, the callback can still fire on mount due to React's deliberate double-invocation behavior.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Telegram Proxies Use Obfuscation and Fake TLS to Evade State Censorship

Deep Packet Inspection (DPI) systems used by countries like Russia and Iran block Telegram by detecting the distinctive byte signatures of its MTProto transport protocol. To counter this, MTProto proxies employ three layered techniques: XOR obfuscation, random padding, and FakeTLS. XOR obfuscation replaces recognisable magic bytes with random data, while random padding alters packet sizes to prevent statistical fingerprinting by DPI systems. FakeTLS goes furthest by wrapping the entire proxy connection in a legitimate-looking TLS handshake, complete with a real server certificate and standard cipher suites, making the traffic indistinguishable from normal HTTPS browsing. Together, these layers ensure that state-level filters see only encrypted TLS records rather than any Telegram-specific patterns.

0 comments Read more at DEV Community