Greenfinger 2.0 turns a single URL into a distributed, searchable web archive

·1 views

Greenfinger 2.0 is an open-source, distributed web crawler built for the JVM that crawls an entire site from a single starting URL and stores every page, image, and version. It produces three output layers simultaneously: a raw file store, a full-text Lucene or Elasticsearch index, and a vector index supporting semantic and image-based search. The system requires no central coordinator or queue service — each node runs the same JAR and claims its own share of URLs, allowing nodes to join or leave mid-crawl without interruption. If the index or vector store needs rebuilding, it can be regenerated from the saved files without re-crawling the site. The project is available on GitHub and can be running in about 60 seconds via a provided CLI and optional Docker setup.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer Argues Algo Trading Edge Lies in Skipping Most Slots, Not Filling Them

A developer publishing a multi-part series on algorithmic trading strategy argues that a healthy prediction-market engine should skip the majority of available 5-minute trading slots rather than treat each one as a mandatory order. The post outlines specific conditions that make a slot 'illegal' to trade, including stale TWAP data, missing reference prices, ask/bid prices outside a 45–55 cent band, and insufficient time within the slot window. The author contends that engines logging only fills are recording accidents, not strategy, and that skips should outnumber fills on a well-functioning trading day. Code examples show that the signal layer is explicitly designed to return a 'none' side and log a skip reason as a successful outcome. The post is framed as part of a broader series and is accompanied by links to the author's Polymarket activity and GitHub repository.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Avatar RS brings full-featured avatar components to Yew, Dioxus, and Leptos

A new open-source library called Avatar RS has been released for Rust-based WebAssembly frameworks, including Yew, Dioxus, and Leptos. The library provides ready-to-use avatar components that handle image loading, fallback text or icons, size variants, color themes, and accessible markup out of the box. It is built on top of Image RS, a WASM-native image primitive that adds lazy loading via the Intersection Observer API, blur-up placeholders, and responsive image support. A Group component allows developers to stack multiple avatars with automatic overlap and an overflow badge that calculates and displays surplus counts without manual configuration. The same API surface and behavior is available across all three supported frameworks, aiming to eliminate repetitive boilerplate code in WASM application development.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How offloading UI calculations to background threads fixed a real-time trading app freeze

A mobile developer at a trading firm encountered severe UI freezes during peak hours after expanding real-time WebSocket data from 10 to up to 137 price levels per stock. The app received around 30 data packages per second, and seemingly minor operations like number formatting, color calculation, and localization overwhelmed the main thread at that volume. An initial attempt to fix performance using Diffable Data Source backfired, as the feature is optimized for partial updates — not full list replacements — making overhead worse. Profiling revealed the true bottleneck: computed properties being evaluated on the main thread for every update cycle. The fix involved moving all calculations to background threads during WebSocket response parsing, leaving the UI layer to simply display pre-computed values, which restored smooth scrolling under peak load.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Builds Home Mini PC Setup to Log Meals, Tasks, and Projects via AI

A developer has set up a personal AI assistant on a home mini PC using OpenClaw, LifeOS, and n8n to manage daily tasks, health logs, and project tracking. The system allows the user to dictate notes in plain language, which the assistant converts into structured database entries stored in SQLite. A simple breakfast log — chicken breast, milk, and dragon fruit totalling roughly 450 kcal — was captured as the first record with a single voice command from a phone. The setup is designed to avoid the fragmentation of juggling multiple apps, with each script given a narrow, specific role such as capturing or editing records rather than broadly modifying files. The developer emphasises accuracy over convenience, preferring to preserve uncertainty in a record rather than allow the assistant to invent missing details.

0 comments Read more at DEV Community