HTTP-only scraping beat headless browser 30–50x in speed on same Shopify page

·1 views

A developer benchmark on a 1 vCPU / 2 GB RAM machine compared HTTP-only scraping against a headless Chromium browser on the same Shopify storefront. The HTTP approach fetched 50 structured product records in 0.14–0.23 seconds using Shopify's public /products.json endpoint, while the browser method took over 7 seconds and consumed 146 MB of RAM for fewer, unstructured results. The headless run also initially failed entirely because Playwright's recommended 'networkidle' wait strategy timed out at 30 seconds, a common issue on modern e-commerce sites with persistent background requests. The test highlights that public JSON or API endpoints — which many platforms expose even without formal documentation — can deliver cleaner, faster data without the overhead of browser rendering. The authors note headless browsers remain necessary when data only exists after JavaScript execution, but recommend checking for API endpoints first.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer builds cookie-free lead attribution for static Nuxt sites without consent banners

A developer running a static Nuxt site with cookieless analytics built a lightweight first-touch attribution system to identify where contact form leads originate. The solution uses localStorage instead of cookies, capturing the external referrer, landing page path, and UTM parameters the first time a visitor arrives. Because the data is written only once, returning visitors are still attributed to their original source rather than a later direct visit. The client-side Nuxt plugin avoids server-side rendering conflicts, and a try-catch wrapper prevents localStorage errors in private browsing from breaking the page. On form submission, the stored source data is sent with the contact payload and formatted server-side into a readable label for notification emails.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Kubernetes Day 3: Understanding File Permissions and Bind Mounts in Docker

File permissions in Linux can be managed using chmod, while the umask command sets default permissions — for example, a umask of 0002 results in 664 permissions for files. Bind mounts in Docker link a host directory to a container directory, functioning as a volume for persistent data sharing. The docker run -v flag is used to map a local folder to a path inside the container, such as ./data to /home/data. Docker images are read-only and cannot be modified in place; any changes result in the creation of a new image layer. A container is essentially a running instance of a Docker image, spun up through the Docker engine.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

C# Tip: Using JsonConverter for Targeted and Global JSON Data Formatting

A practical C# guide demonstrates how to build a custom JsonConverter that capitalizes the first letter of string values during JSON deserialization. The converter, named UpperCaseFirstCharConverter, is designed for data-import scenarios where incoming JSON fields like names may have inconsistent casing. Developers can apply it globally via JsonSerializerOptions or register it in ASP.NET Core's Program.cs to cover all serialization operations. However, global application affects all string fields, including unintended ones like state codes, which can cause data errors. The recommended fix is to apply the converter selectively using the JsonConverter attribute directly on specific model properties such as FirstName and LastName.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Prompt Engineering Explained: From Zero-Shot to RAG and Beyond

Prompt engineering is the practice of designing inputs for large language models to extract more accurate and reliable outputs without modifying the model itself. A technical overview published on DEV Community traces the evolution of six key prompting techniques, arranged by the complexity of problems they address. The progression begins with zero-shot prompting, where a task is described with no examples, and advances through few-shot learning, Chain-of-Thought reasoning, and self-consistency validation. More sophisticated techniques include Retrieval-Augmented Generation, which grounds responses in external or private data, and Automatic Reasoning and Tool-use, which enables models to invoke real-world tools. The article argues that understanding these techniques in sequence helps clarify why each one emerged as a solution to the limitations of its predecessor.

0 comments Read more at DEV Community