Developer Builds Tool to Convert Raw Web Pages into Clean JSONL Datasets for LLM Fine-Tuning

A developer has created an automated pipeline that transforms seed URLs into clean, token-aware JSONL datasets suitable for fine-tuning large language models on domain-specific tasks such as legal, medical, or financial content. The tool performs same-domain crawling, strips boilerplate elements like navigation bars and ads, and removes duplicate content using both exact hashing and near-duplicate detection via 6-gram Jaccard similarity. Text is then split into coherent chunks at paragraph boundaries, with each record retaining provenance metadata including source URL, hostname, domain, title, and chunk index. Testing on three US Supreme Court opinions from Cornell LII produced 443 clean chunks totalling around 173,000 estimated tokens with zero duplicates. The developer noted key limitations, including incompatibility with JavaScript-rendered pages and WAF-protected sources, and flagged that near-deduplication scales as O(n²), requiring caps on large crawls.