Developer Builds HashDup, a Memory-Safe Duplicate File Finder CLI in Node.js

·45 views

A developer has released HashDup, an open-source command-line tool built with native Node.js to find and report duplicate files on a system. The tool addresses two common problems with naive duplicate-detection scripts: wasted CPU and disk I/O from hashing every file, and out-of-memory crashes when loading large files into RAM. HashDup uses a two-phase pipeline that first filters files by size using lightweight metadata checks, discarding unique-sized files before any content is read, then applies chunked SHA-256 streaming hashes only to size-matched candidates. By processing files in 64KB buffers rather than loading them whole, the tool keeps memory usage constant at around 30MB regardless of file size. HashDup is available under the MIT license and can be run immediately via npx from the GitHub repository at github.com/mahdyarmonfared/hashdup-cli.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How HMAC Request Signing with Timestamps Hardens API Security

Bearer API keys alone let an intercepted request be replayed indefinitely, tampered with, or reused against other endpoints. The article proposes request signing, where each call carries an HMAC-SHA256 signature computed over the method, path, body hash, and a timestamp. Servers reject requests whose timestamps fall outside a 30-second window and verify signatures with constant-time comparison to prevent timing attacks. The author provides Python client and FastAPI server implementations and stresses that deterministic JSON serialization, such as sorted keys and fixed separators, is necessary for signatures to verify correctly. The approach mirrors techniques used by AWS SigV4, Stripe webhooks, and GitHub webhooks.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Three-Stage Filter Stops False Golf Cart Geofence Alerts Near Course Edges

Naive geofence alerts for golf carts fire repeatedly when GPS jitter near the green's edge crosses the boundary, training riders and supervisors to ignore them. The proposed fix stacks three filters: discard fixes with poor reported accuracy, treat a 5 m band around the zone edge as undecided, and require a new state to persist for 10 seconds. These are implemented as a small Python state machine with a companion test file. Each filter alone is insufficient, but together they change the alert only when the evidence is sustained.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer launches VibePost, an AI platform for social media content creation

An independent developer has built VibePost, an AI-powered platform aimed at simplifying social media content creation. The tool is designed to bring idea generation, post writing, editing, and publishing schedules into a single workspace instead of spreading them across multiple apps. The builder notes that creating content consistently is often more time-consuming than expected, which motivated the project. VibePost is an independently built product under JavaSpace and is still being refined, with its creator seeking feedback from developers and indie hackers.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Pippo, a locally run AI agent, introduces itself and its freelance experiments on DEV

Pippo, a self-described AI agent that runs entirely on a local Windows PC rather than in the cloud, has introduced itself on DEV Community. It writes and tests code in Python, C++, JavaScript/TypeScript and shell, and is experimenting with CAD work in Fusion 360 and Blender. The agent says it is attempting to earn income by bidding on on-chain marketplaces such as OpenTask and TaskMarket and delivering work in USDC, though its first payment was still pending when the post was written. It plans to document its engineering work, earnings and field observations on a public blog.

0 comments Read more at DEV Community