Developer Open-Sources Harvest, a Self-Healing Web Scraper Powered by LLMs

·1 views

A developer has released Harvest, an open-source web scraper designed to automatically repair itself when target websites change their HTML structure. The tool uses a large language model to detect broken selectors, generate replacements, and validate them against a schema — storing the full selector history locally for rollback. Harvest also implements semantic caching, storing extraction results by meaning rather than exact query text, which the developer reports reduces LLM token consumption by 50–70%. Additional features include structural DOM diffing to highlight page changes and a script generator that performs a one-time LLM analysis to produce reusable, zero-token scraping scripts. The project is available on GitHub and can be installed via pip, with support for Cloudflare bypass and an MCP server interface for integration with AI agents.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer Builds AI Career Guidance Platform Using Google Gemini for Students

A developer has created Supernova AI, an AI-powered career guidance platform, as a capstone project for the Google Cloud Gen AI Academy APAC 2026. The platform addresses the challenge of career confusion faced by students and professionals who have access to abundant information but lack personalized direction. Users input their educational background, skills, interests, and career goals, after which the system generates a tailored Career Report with a match score, recommended path, skill gaps, and a learning roadmap. Built using Google Gemini and an Express.js backend, the tool is designed to act as a decision-support system rather than a replacement for human judgment. The project was motivated by the creator's own experience growing up with limited exposure to diverse career options, particularly in underserved communities.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Laravel Cloud $5 Plan vs $5 VPS: What the Real Bill Looks Like

Laravel Cloud revamped its pricing in June 2026, introducing a $5/month Starter tier with scale-to-zero hibernation and spending caps designed to make low-traffic apps affordable. Unlike a flat-rate $5 VPS, the Starter plan is usage-metered, meaning the monthly fee is a floor rather than a fixed cost. Apps with continuous queue workers, frequent scheduled tasks, or persistent databases can accumulate charges around the clock, undermining the hibernation benefit. The $5 tier genuinely suits idle workloads like portfolio sites or infrequent demos, but production Laravel apps with Horizon, schedulers, and dedicated databases often face higher bills than a comparable VPS. The core takeaway is that usage-metered platforms excel for sporadic traffic but offer limited savings over flat-rate hosting for steady, always-on workloads.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Open-Sources Self-Healing Web Scraper That Auto-Fixes Broken Selectors

A developer has released Harvest, a free, MIT-licensed web scraper built to solve a common frustration: scrapers breaking whenever a website updates its HTML. When selectors fail due to site changes, Harvest uses a large language model to automatically regenerate and validate new CSS selectors, storing the full history locally for rollback. The tool also features a meaning-based semantic cache that can reduce token usage by 50–70% by recognising similar prompts, and a script generator that produces standalone Python scrapers requiring zero LLM tokens at runtime. Harvest runs as an MCP server, making it compatible with AI agents like Claude and Cursor, and includes built-in Cloudflare bypass via Chromium. The project, comprising roughly 9,500 lines of Python, was built in three weeks and is available on GitHub, though it currently has no established community.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why AI Agents Forget Everything — and How Developers Can Fix It

Most AI agents lose all context the moment a session ends because they rely solely on a temporary context window with no persistent memory layer. Developers distinguish four memory types — working, episodic, semantic, and procedural — but most early agent frameworks only implemented the first. Simply expanding the context window does not solve the problem, as it raises costs, introduces 'context rot' from irrelevant history, and still resets after each session. Frameworks such as Mem0, Zep, Letta, and Cognee are now purpose-built to add the missing layers, with multi-agent systems requiring a shared persistent memory source so all agents operate from a common ground truth. The urgency is growing: Deloitte's 2026 outlook projects that nearly half of companies already using generative AI will run agentic AI pilots or production deployments by 2027, roughly double the 2025 figure.

0 comments Read more at DEV Community