Developer releases duckdiff, a Python tool for comparing large structured data files

·1 views

A software developer has released duckdiff, an open-source Python package designed to compare large structured data files such as CSV, TSV, and Parquet formats. The tool was built to address limitations of existing solutions like Excel, standard diff utilities, and pandas, which struggle with files exceeding memory limits or require sorted rows. Powered by DuckDB, duckdiff streams data off disk and supports N-way comparison of multiple sources in a single pass, rather than repeated pairwise checks. It also offers features such as fuzzy column mapping, schema intersection, and a key-detection subcommand to identify unique row identifiers. The package is available on PyPI at version 0.1.0, includes 161 tests, and its source code is hosted on GitHub.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Vercel Sandbox Offers Isolated MicroVMs to Safely Run AI-Generated Code

Vercel Sandbox runs untrusted code, including AI-generated scripts, inside ephemeral Firecracker microVMs that are fully isolated from the host application's process, filesystem, and environment variables. Each sandbox is created on demand, executes the required commands, and is discarded afterward, preventing any persistent access or state leakage. Unlike running code via child_process or eval within a Vercel Function, the sandbox approach ensures that compromised or malicious code cannot access API keys, database URLs, or other secrets. Developers can configure per-sandbox timeouts and CPU limits, and expose running servers on public URLs for live previews of AI-generated components. Vercel Sandbox is intended specifically for untrusted, externally authored code rather than first-party build or CI pipelines.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Guillotine vs Free Nesting: Choosing the Right Sheet Cutting Method

When laying out rectangular parts on sheet material, two main cutting strategies exist: guillotine cutting, which uses full straight cuts across the entire sheet, and free nesting, which places parts more flexibly to reduce waste. Guillotine cutting suits machines like panel saws and table saws where strip-based workflows and simple cut sequences are prioritized over maximum material efficiency. Free nesting can minimize offcuts but may introduce complex or awkward cuts depending on the equipment being used. The choice should factor in material cost, machine capability, and the operator's ability to execute a safe cutting sequence. A free browser tool called CutListEngine allows users to compare both methods using the same inputs without requiring a signup or file upload.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Choose the Right Linux Distribution Based on Your Use Case

No single Linux distribution suits every user, as the best choice depends on individual needs, experience level, and intended tasks. Beginners are advised to start with user-friendly options like Ubuntu, Linux Mint, or Zorin OS, while developers may prefer Fedora or Arch Linux for access to cutting-edge tools. Gamers can turn to distributions such as Bazzite or Pop!_OS, and cybersecurity professionals commonly rely on Kali Linux or Parrot OS for penetration testing. Server administrators are pointed toward stable options like Debian and Rocky Linux, while lightweight distributions such as Lubuntu and Linux Lite can breathe new life into older hardware. Experts recommend basing the choice on actual usage requirements rather than popularity, and suggest testing options via a Live USB before committing to a permanent installation.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Cloudflare silently blocked ChatGPT, Perplexity and Claude bots for three weeks

A developer discovered that Cloudflare's 'Block AI Bots' security setting had been silently returning HTTP 403 errors to AI search crawlers — including OpenAI's SearchBot, PerplexityBot, and Claude-SearchBot — for three weeks, while standard search engines like Googlebot passed through unaffected. Because the blocks occurred at the edge before requests reached the origin server, Apache access logs recorded nothing, making the problem invisible to conventional monitoring tools. The developer only uncovered the issue by manually simulating each bot's user agent with curl against the live public URL, bypassing local network shortcuts. A secondary problem was also found: fail2ban was misreading Cloudflare relay IPs as malicious clients and banning them, which periodically caused HTTP 521 errors for all visitors. The developer responded by building a daily automated crawler check that probes the real public URL across 16 user agents and alerts on any unexpected status change.

0 comments Read more at DEV Community