SShortSingh.
Back to feed

How Elasticsearch Finds Results in Milliseconds: Inverted Indexes and Shard Routing

0
·2 views

Elasticsearch achieves fast full-text search by using an inverted index, which maps words to documents rather than scanning documents for words, enabling direct lookups instead of linear reads. At indexing time, documents are broken into tokens and stored in posting lists, allowing multi-word queries to quickly intersect or union sorted lists. To handle billions of documents, the index is split into shards — each a self-contained inverted index — distributed across multiple nodes so storage and query load are shared. A document is assigned to a shard by hashing its ID against the shard count, meaning reads by ID target a single shard, while full-text queries scatter to all shards and a coordinating node merges the ranked results. These design choices come with trade-offs: indexing is heavier than a simple database write, search is near-real-time rather than instant, and the shard count must be fixed early since changing it breaks the routing hash.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Developer releases production-ready MCP server template built around six core design principles

A developer has published a reusable template for building production-grade Model Context Protocol servers, addressing the gap between basic tutorials and real-world deployment. The template supports both stdio and HTTP transports using a factory pattern that creates a fresh server instance per request, enabling stateless horizontal scaling. All configuration and tool inputs are validated through Zod schemas at startup, while logging is strictly routed to stderr to avoid corrupting the JSON-RPC stream on stdout. Security measures include an SSRF-aware URL-fetching tool with hostname allowlisting, byte caps, and timeouts. The full package, called MCP Quickstart Kit, also includes Docker, CI setup, and a Stripe integration for monetising tools.

0
ProgrammingDEV Community ·

Better Auth Open Source Framework Acquired by Vercel to Expand Auth Development

Vercel has announced that Better Auth, the open source authentication framework, is joining the company. The acquisition aims to provide the Better Auth team with greater resources while keeping the framework independent and compatible across different platforms. Better Auth previously gained attention for addressing multi-tenant authentication challenges in Next.js and for taking over stewardship of Auth.js and NextAuth.js. Looking ahead, the team plans to focus on authentication solutions for AI agents, enabling secure and revocable user-delegated access in modern applications.

0
ProgrammingDEV Community ·

Agentic AI Loops Are Replacing One-Shot Prompting in Software Development

A growing perspective in AI-assisted development argues that single prompts are no longer sufficient for serious agentic workflows. Instead of manually re-prompting an AI agent after each failure, developers can define iterative loops that run autonomously until a goal — such as passing all CI checks — is met. Tools like Anthropic's Claude Code support this model through primitives such as goals, loops, schedules, and reusable skills. The shift moves a developer's role from crafting individual instructions to defining success criteria, verification methods, and stopping conditions. Proponents argue the loop-based paradigm applies broadly across agentic coding tools, not just Claude Code.

0
ProgrammingDEV Community ·

How Developers Can Test Twilio Webhooks Without Deploying to Staging

Developers building Twilio integrations for SMS or voice callbacks face a core challenge: Twilio's servers cannot reach a local machine at localhost, making webhook testing difficult without a public URL. A common workaround is setting up a minimal Express server that accepts Twilio's form-encoded POST requests and verifies each one using HMAC-SHA1 signature validation. To expose the local server to Twilio, developers typically use HTTP tunneling tools such as ngrok, though its free tier assigns a new URL on every restart, requiring manual webhook URL updates in the Twilio console each session. Alternative tools like Anonymily aim to solve this by offering a stable, named endpoint that persists across restarts, reducing friction in the local development workflow. The overall goal is to enable faster debugging and payload inspection without the slow feedback loop of repeatedly deploying to a staging environment.