SShortSingh.
Back to feed

Scraping product data is easier by reading embedded JSON, not parsing HTML

0
·2 views

Web developers building product scrapers often rely on CSS selectors, which break whenever a site undergoes a redesign. A more reliable approach involves extracting JSON data already embedded in the page, available in three main forms: schema.org JSON-LD script tags, framework state blobs like __NEXT_DATA__ or __NUXT__, and internal XHR API calls made by the page itself. JSON-LD offers a standardized product shape usable across retailers but can be inaccurate for real-time stock status. Framework state blobs typically contain richer data including variant details and inventory, while direct API calls — as used with SHEIN's mobile API — bypass HTML rendering entirely. Using these JSON sources reduces dependence on headless browsers and makes scrapers significantly more resilient to frontend changes.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Why cron jobs silently fail: the case for receipts over stale labels

A software engineer running agent infrastructure on a cron schedule discovered a class of silent bugs caused by trusting stale metadata labels written by earlier processes. The core issue is that labels are assertions about the world at the time of writing, not verifiable observables that can be re-checked at read time. In one case, a label identifying a search backend remained unchanged after the backend rotated, causing a subsequent session to reason incorrectly for twenty minutes without any error or crash. The author argues that cron systems are especially vulnerable because the writing process, the reading process, and the scheduler all operate independently with no shared responsibility for detecting staleness. The proposed fix is to embed observability directly into stored values — including timestamps and TTLs — so readers can compute freshness rather than blindly trust what was written.

0
ProgrammingDEV Community ·

Bitcoin's First Price Was $0.000764 — Set by One Man and His Power Bill

On October 5, 2009, a pseudonymous user called NewLibertyStandard published Bitcoin's first ever exchange rate, valuing one dollar at 1,309.03 BTC, calculated purely from U.S. residential electricity costs and network mining output. A week later, on October 12, early Bitcoin contributor Martti Malmi completed the currency's first documented cash trade, selling 5,050 BTC to NewLibertyStandard for $5.02 via PayPal — a sum that would be worth over half a billion dollars at recent prices. NewLibertyStandard's correspondence with Bitcoin creator Satoshi Nakamoto from that same period reveals a far more mundane reality, with messages focused on Linux build errors and a laptop fan spinning at full speed from mining. Despite anchoring the economics of what would become a multi-trillion-dollar asset class, his contributions went largely unrecognised and undocumented at the time. The 1,309.03 BTC that original one dollar purchased would be valued at roughly $84 million today.

0
ProgrammingDEV Community ·

Developer Adds GeekDock as Free Bonus to QuietWare macOS App Bundle for 24 Hours

Independent macOS developer has added GeekDock as a no-cost bonus to the QuietWare Workspace Collection for a limited 24-hour window. The bundle includes five productivity apps: ScreenShelf, FocusForm, File Fetch, PopNote, and GeekDock, covering file organization, workspace management, and Dock customization. All apps are priced as one-time purchases with no subscriptions or account requirements. The complete five-app collection is available for $9.99 until the promotion expires.

0
ProgrammingDEV Community ·

Ten Lesser-Known CSS Animation Features That Can Elevate Your UI

CSS has gained several powerful animation capabilities in recent years that remain largely absent from mainstream tutorials. The @property rule allows developers to register typed custom properties, enabling smooth gradient and conic-effect animations without JavaScript. A newer scroll-driven animation API lets elements animate in sync with scroll position or viewport visibility using animation-timeline, eliminating the need for scroll listeners or Intersection Observers. The offset-path property enables elements to travel along arbitrary curved paths, with offset-rotate automatically orienting them along the curve. These features collectively reduce reliance on JavaScript for motion effects and offer more declarative control over UI animations.