Developer Integrates Scrapling MCP Server to Give Claude Code Live Web Access

·1 views

A developer has connected the open-source Scrapling web-scraping framework to Claude Code via its built-in Model Context Protocol (MCP) server, enabling the AI agent to fetch real web pages that bypass modern anti-bot defenses like Cloudflare. The integration exposes ten scraping tools to Claude Code, including standard HTTP fetches, headless-browser fetches using Playwright, stealth fetches via patchright, and screenshot capture. The setup was installed locally on Windows using a dedicated Python 3.14 virtual environment, requiring roughly 700 MB of browser binaries for full rendering and stealth capabilities. To reduce token consumption, the tools default to returning markdown with only main content, which the developer reports cut payload size by around 71 percent on content-heavy pages. The developer credited the core scraping engine entirely to the Scrapling maintainer, describing their own contribution as the local installation and MCP wiring that makes those capabilities available to the agent.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

llms.txt: The Markdown File That Helps AI Crawlers Navigate Your Website

An llms.txt file is a plain Markdown document hosted at a website's root domain that gives AI crawlers a structured, curated overview of a site's most important pages. The concept was proposed by Jeremy Howard in 2024, based on the idea that language models perform better with clean, contextual summaries than with raw HTML. A standard llms.txt file includes a site name, a brief blockquote summary, optional notes, and categorised links with short descriptions. Unlike robots.txt or a sitemap, it is written specifically for machine readers rather than search engine indexing or human navigation. No major AI provider has confirmed that the file directly influences rankings or citations, but proponents argue it is a low-cost, low-risk practice that reduces ambiguity for AI models parsing web content.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer builds repeatable PDF research pipeline after 48-hour literature review struggle

A developer working on an academic PDF annotation tool in March 2025 spent 48 hours manually sifting through research papers before recognising the need for a structured, automated workflow. The project involved scraping PDFs to extract text coordinates for LayoutLM model experiments, but brittle parsing scripts and disorganised citation tracking slowed progress significantly. To fix this, the developer layered multiple tools covering automated discovery, structured fact extraction, and auditable citation trails, reducing an eight-hour literature skim to around 40 minutes. The build also hit a CUDA out-of-memory error during LayoutLM fine-tuning, which was resolved by switching to streamed data loading and gradient accumulation. The resulting pipeline is presented as a practical, scalable approach for anyone needing deeper research insights faster than manual reading allows.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How API Fixture Patterns Can Close the Email Regression Testing Gap

Engineering teams commonly test API responses via status codes and JSON schemas but frequently overlook verifying the outbound emails those APIs trigger. This gap can cause real user-facing failures — such as broken invite links or invalid tokens — even when the API itself returns a successful response. A practical fix involves treating outbound email as a fixture-backed contract: creating a disposable per-run inbox, triggering the API action, and asserting the resulting message against a compact set of expected fields. The fixture checks stable elements like the recipient, subject prefix, sender domain, and key call-to-action paths rather than snapshotting the entire email body. Privacy guardrails remain important throughout, including keeping test inboxes isolated from production traffic and limiting how much message data is captured.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer builds instant filesystem rollback tool to rein in rogue AI agents

A developer testing an autonomous AI coding agent watched it corrupt an entire local workspace after the agent hallucinated a flawed fix and executed a destructive terminal command. While sandboxing solutions like Docker can limit damage to the host machine, they do not preserve mid-task state, forcing users to restart multi-step workflows from scratch after any failure. The developer argued that autonomous agents need a built-in checkpointing primitive similar to Git commits or VM snapshots, taken automatically before every shell command is executed. To keep the process fast, they implemented a solution using Linux's OverlayFS, which separates the agent's writes into an isolated scratch layer while keeping the original files untouched, making each checkpoint a sub-20-millisecond operation. Successful commands are committed by syncing the scratch layer, while destructive ones can be instantly rolled back by discarding it entirely.

0 comments Read more at DEV Community