Binary search on rank data cuts 4-million-item crawl from a week to a day

·1 views

A software engineer needed to crawl roughly 4 million items from a public registry within a single day using parallel workers. Splitting the keyspace evenly by alphabet failed because key distribution was uneven, leaving one shard carrying 71% of the total workload. The engineer instead used the registry's listing API, which returns a key's global rank in the sorted order, as a binary search oracle. Running about 40 API calls per boundary, the algorithm identified cut points where each shard held an equal share of items regardless of alphabet width. The resulting balanced partition reduced the fattest shard from 71% of the work to roughly 1/N, turning a projected week-long crawl into a one-day job.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Rise Reforming Converts Landfill Biogas into Chemicals Using Modular On-Site Tech

Rise Reforming, a YC S26 startup founded by George, Lucas, and Jona, is developing modular technology to convert biogas from landfills, farms, and wastewater plants into valuable industrial chemicals. The company acts as an intermediary, purchasing biogas from producers and selling the resulting chemicals to end users. It is initially targeting dimethyl ether for the cosmetics industry before scaling toward methanol, a widely used industrial chemical. The startup aims to address both the fragility of centralized, fossil-fuel-dependent chemical supply chains and the widespread underutilization of biogas, roughly 60% of which is currently burned for low-margin energy or flared. The U.S. generates around 780 billion cubic feet of biogas annually, representing an estimated $20 billion per year in potential methanol output.

0 comments Read more at Hacker News

ProgrammingDEV Community ·

Refactor or Rewrite? A Practical Framework for Engineering Teams

Engineering teams often debate whether to refactor existing code or rewrite it entirely, a choice driven by risk, economics, and time rather than emotion. Refactoring is generally recommended when a system is stable and functional but locally messy, while a rewrite is justified only when the core architecture fundamentally blocks progress. Before committing to a rewrite, teams must deeply understand the existing system, since legacy code often contains undocumented business logic that a rushed rewrite will simply lose. Rewrites also demand significant time and resources, requiring a deliberate business decision rather than a developer's frustration in the moment. In both cases, investing in tests first is essential, as neither refactoring nor rewriting is safe without a reliable safety net.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Lossless Compression Techniques Can Boost RAG Agent Performance and Cut LLM Costs

Large Language Models operate within a fixed token limit called a context window, which poses a challenge for Retrieval Augmented Generation (RAG) agents that must inject retrieved documents into that space. When retrieved content exceeds the context limit, agents typically resort to truncation or summarization, both of which risk losing critical information or introducing inaccuracies. Lossless compression offers an alternative by reducing the token footprint of retrieved text without discarding any factual content, preserving full fidelity for accuracy-sensitive domains like law and medicine. Techniques such as keyword extraction, deduplication, semantic chunking, and syntactic simplification can be applied before or after retrieval to maximize information density. Beyond improving response quality and reducing hallucinations, these methods can also lower operational costs since most LLM APIs charge based on token usage.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Monorepo vs Polyrepo: Lessons from Meta, AWS, and a 15-Repo Migration

A software engineer who worked at both Meta and AWS shares firsthand observations on how each company manages code organization at scale. Meta's single Mercurial monorepo enabled cross-cutting changes in one commit and made code discoverable, but demanded distributed build tooling and careful dependency management. AWS's polyrepo model offered clear team ownership and independent deployments, yet led to duplicated infrastructure patterns and poor discoverability across thousands of repositories. Drawing on both experiences, the engineer migrated SID Technologies from 15 separate repositories into a single monorepo over a weekend. The key takeaway is that the monorepo versus polyrepo choice is fundamentally an organizational decision rather than a purely technical one.

0 comments Read more at DEV Community