Web Sweeper Builds Auditable, Provenance-Protected Libraries for AI Training

·1 views

Developer Christian Cassarly has released Web Sweeper, an open-source tool built under Jesus New OS aimed at creating trustworthy digital libraries for future AI training and research. The system moves content through a structured pipeline covering source discovery, policy screening, acquisition, staging, publication, and live verification. Key features include provenance preservation, duplicate protection, resumable processing, and permanent verification receipts to ensure full auditability. Rather than simply downloading files, Web Sweeper focuses on producing reliable, traceable datasets that AI systems and researchers can use responsibly. The project is actively under development, with source code and documentation publicly available on GitHub.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer Rethinks AI Agent Memory: The Real Problem Is Knowing What You Want

In the final part of a three-part series on DEV Community, a developer reflects on rebuilding an AI coding agent harness after earlier failures, trimming subagents and consolidating responsibilities to let the model perform at its best. The core insight reached is that effective agent memory is not just about storing and retrieving information well, but about delivering the right context at precisely the right moment in a workflow. The author argues that a perfect initial prompt is no longer realistic, since tasks grow in complexity and requirements evolve, making mid-process intervention essential. To address this, the developer proposes a system where team preferences and project context are continuously collected and injected into the agent at relevant stages, either through automated hooks or on-demand reads. This approach, described as 'stages,' aims to close the gap between what a user actually wants and what the agent produces, without turning memory management into an added burden.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Browser tool reads your own image and file bytes locally, no upload needed

A developer has built eleven browser-based pages that parse file formats — including JPEG, PNG, GIF, PDF, ZIP, and fonts — entirely within the browser tab without uploading any data. Each page lets users drop their own files rather than relying on author-selected examples, shifting the burden of proof from curated demos to real-world inputs. The project addresses a common flaw in technical demonstrations, where cherry-picked specimens can hide a technique's weaknesses. Building the tool created three distinct engineering challenges, including writing robust parsers that handle malformed or corrupt files from real user disks. The pages also print carefully worded conclusions about their results, designed to avoid overstating what the parsed data can actually prove.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer builds fully client-side semantic search for 796 pages using no server or AI calls

A developer at artwaste.land built a fully browser-based semantic search engine for their 796-page static website, hosted on Cloudflare, without any server, vector database, or runtime AI model. The system was driven by a strict site rule prohibiting user input from being sent to third parties, which ruled out standard embedding API approaches. Instead of shipping a full transformer model to the browser, the team used a Model2Vec technique — running the model once at build time to produce a static word-vector lookup table distilled from Xenova/all-MiniLM-L6-v2. The entire search engine consists of three static JSON files totalling under 5.4 MB compressed and 401 lines of vanilla JavaScript with no external dependencies. The approach trades some accuracy for zero query-time latency, no per-call costs, and complete user privacy.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Auto-Seed a Postgres Database with a Single Docker Compose Command

Developers using Docker Compose can reliably reproduce a Postgres database environment with one command, but populating it with seed data has traditionally required a separate manual step. A workflow combining Docker Compose healthchecks and a tool called Seedfast aims to fold database seeding directly into the `docker compose up` process. The key decisions involve choosing where the seed command runs — either in the Postgres init directory, as a one-shot Compose service, or from the host shell — and ensuring it only fires after Postgres is fully ready to accept connections. The `pg_isready` healthcheck bridges the gap between a container reporting as 'running' and the database actually being ready, preventing failed seed attempts due to refused connections. Schema bootstrapping belongs in the first-boot init directory, while frequently changing test data is better handled by a healthcheck-gated seed step that can be re-run on demand.

0 comments Read more at DEV Community