SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Developer Builds HashDup, a Memory-Safe Duplicate File Finder CLI in Node.js

0
·5 views

A developer has released HashDup, an open-source command-line tool built with native Node.js to find and report duplicate files on a system. The tool addresses two common problems with naive duplicate-detection scripts: wasted CPU and disk I/O from hashing every file, and out-of-memory crashes when loading large files into RAM. HashDup uses a two-phase pipeline that first filters files by size using lightweight metadata checks, discarding unique-sized files before any content is read, then applies chunked SHA-256 streaming hashes only to size-matched candidates. By processing files in 64KB buffers rather than loading them whole, the tool keeps memory usage constant at around 30MB regardless of file size. HashDup is available under the MIT license and can be run immediately via npx from the GitHub repository at github.com/mahdyarmonfared/hashdup-cli.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Why Workspaces Should Require Both DNS Verification and Email Confirmation in 2026

Developers building workspace onboarding systems are advised to treat DNS domain verification and email confirmation as two distinct, complementary security checks rather than interchangeable ones. DNS TXT verification proves that an organisation controls a domain namespace, while email confirmation proves only that an individual can access a specific mailbox at a given moment. Neither check is permanent, so engineers are encouraged to log both as timestamped evidence events rather than storing a simple verified flag in a database. This approach becomes especially important in sectors like healthcare, where a clinic acquisition can leave former staff with lingering mailbox access even after ownership of the domain has changed hands. Combining both verification methods with auditable records narrows the identity claim and allows any future disputes or access anomalies to be traced back to a specific policy version and observed timestamp.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

MCP Protocol Sparks Developer Debate Over Token Costs and Context Bloat

A blog post titled 'Why MCP Was Always a Bad Idea' has reignited debate over the Model Context Protocol, arguing it was designed for 2024-era AI models and imposes unnecessary token costs and context bloat. The post gained traction on Hacker News, attracting around 165 upvotes and over 100 comments within roughly 13 hours, while also drawing divided reactions on X. Critics contend that MCP tool definitions, schemas, and server handshakes consume valuable context on every call, driving up costs without proportional benefit. Defenders counter that MCP's explicit tool boundaries offer meaningful sandboxing and auditability advantages, particularly for agents without full system access. The core disagreement is not whether the protocol works, but whether its overhead is justified given how modern AI agents are actually deployed.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

AI Agent Chat Worked But Memory Retrieval Silently Failed Due to Docker Misconfiguration

A developer investigating a Docker-based AI agent deployment discovered that while chat functioned normally, active memory retrieval was silently failing with timeouts. The root cause was a misconfigured URL: when the required gateway address was absent, the system fell back to a container-internal IP address that the client's tool execution environment could not reach. Passive memory injection continued to work, meaning the assistant still appeared knowledgeable, masking the underlying retrieval failure from the user. The assistant silently fell back to local file reads and searches without surfacing any visible tool error. The issue was traced to a missing configuration field already documented for multi-node deployments, making it a deployment mistake rather than an upstream software bug.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Developer Finds AI Draft Fails to Preserve Project Context Across Sessions

A developer working on a multi-part series about TencentDB Agent Memory integration found that an AI-generated draft, while topically accurate, failed to reflect the accumulated context and personal experience built across four prior posts. The draft correctly covered memory systems concepts but substituted the structure of a reference article for the author's own research narrative, effectively erasing earlier work. The incident prompted the author to articulate a more precise requirement: an AI assistant must distinguish between firsthand debugging experience, official documentation, and external references when continuing a project. This is especially critical during session or agent handoffs, where continuity of project history — not just topic familiarity — determines whether work can meaningfully proceed. The author clarifies the finding was not a benchmarked comparison of memory products, but a real-world prompt that exposed a gap in how AI tools carry forward accumulated project context.

0 comments Read more at DEV Community