Python tool converts HTML pages to Markdown chunks and tracks section-level changes

·3 views

A developer has built a Python tool using only the standard library that converts web pages into clean, structured Markdown for use in search indexes or retrieval-augmented generation (RAG) pipelines. The script parses HTML into a tree, strips navigation and decorative elements, and renders content as CommonMark Markdown with GitHub Flavored pipe tables. It splits the output into named sections based on heading paths, then compares two versions of a page to classify each section as added, removed, renamed, changed, links-only, or unchanged. The approach was demonstrated using Python 3.13 and 3.14 documentation pages fetched from docs.python.org. This allows targeted re-embedding of only modified chunks rather than reprocessing an entire page on each update.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Python tutorial shows how to extract and track product prices from JSON-LD data

A developer tutorial published on DEV Community demonstrates how to monitor product price and stock changes using JSON-LD structured data embedded in web pages. The guide explains that many e-commerce product pages include machine-readable Schema.org markup, which can be parsed programmatically in Python. The tutorial covers extracting key fields such as price, currency, availability, and offer count from both single offers and aggregate offer blocks. It also walks through building a diff mechanism to detect changes between successive data snapshots, such as price drops or stock status updates. Unit tests included in the article validate the parsing logic across various edge cases, including malformed JSON and non-standard data formats.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Builds Privacy-First Chrome Extension to Track Subscriptions Locally

A developer created SubPulse, a Chrome extension for tracking recurring subscriptions, renewals, and spending without requiring an account, bank connection, or email access. All subscription data is stored locally in the browser, with users manually entering details such as subscription name, price, currency, billing frequency, and renewal date. The extension sends Chrome notifications before renewals and displays a badge for upcoming or overdue payments, addressing the common problem of forgotten charges. SubPulse also supports multiple currencies with automatic conversion, spending analytics, budget limits, and backup and import functionality for data portability. Now at version 1.2.5, the extension includes additional features such as calendar and CSV export, dark and light themes, and weekly renewal summaries for Pro users.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Laya AI Decision API Promises Sub-35ms Classification Without Large Language Models

A developer has launched Laya, a free AI decision-making API designed to replace large generative models like GPT-4o for simple classification tasks such as spam filtering, sentiment analysis, and prompt injection detection. Unlike token-by-token generative models that can take 1,500ms to 3,500ms to respond, Laya uses a 322-million-parameter multilingual bidirectional encoder to compute results in a single forward pass, targeting under 35 milliseconds at the model level. The tool offers six dedicated endpoints covering support ticket triage, content moderation, spam filtering, and a universal decision engine, among others. Laya is available via the RapidAPI marketplace, though real-world latency through that gateway ranges from 750ms to 1,200ms due to proxy routing and authentication overhead. The service claims a privacy-first design, processing all payloads in ephemeral RAM with no logging or storage of user data.

0 comments Read more at DEV Community

ProgrammingHacker News ·

Lofi Cities: Browser-Generated Pixel Art Cityscapes Paired With Lofi Music

A developer has launched Lofi Cities, a web-based creative project accessible at loficities.com. The platform combines pixel-art animations of city nightscapes with lofi music generated directly in the browser. The project was shared on Hacker News under the 'Show HN' category, which is used for community-built tools and experiments. It received modest early engagement with 7 points and one comment at the time of posting.

0 comments Read more at Hacker News