How LioranDB Builds a Dedicated Full-Text Search Architecture

·1 views

LioranDB, developed by Swaraj Puppalwar under Lioran Group, uses a dedicated text-segment architecture to handle full-text search queries that standard secondary indexes cannot efficiently process. Text is tokenized by splitting on non-alphanumeric characters, with options to normalize to lowercase and filter stopwords before indexing. Each segment stores structures including a term dictionary, posting lists, a document map, document-length norms, optional term positions, a Bloom filter, and segment metadata. Bloom filters allow the query engine to skip irrelevant segments quickly, while local-to-global document ID mapping keeps posting lists compact. The system supports AND/OR query modes and exposes metrics such as query time, postings read, and segments searched, reflecting its design as a specialized engine distinct from the core document database.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Parts-Based API Design Offers a Unified Way to Handle Multimodal AI Sessions

Multimodal AI APIs are typically documented one modality at a time, leaving developers to figure out how to manage text, images, audio, and video within a single coherent session. A parts-based approach — where each request is an ordered list of typed elements — simplifies interleaving multiple media types without requiring separate correlated API calls. Developers should account for provider-specific payload size limits, as exceeding them often returns an unhelpful error rather than a clear prompt to upload assets by URI instead. Response handling also requires a state machine approach, since text streams token by token while images arrive whole and audio may come in chunks, making uniform treatment of stream events a source of subtle bugs. Multi-turn sessions with images can grow costly quickly, and strategies like summarising older turns or letting users pin specific assets in context help keep token usage and expenses manageable.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Build a Daily Card Feature Without a Database Using Hash Functions

A developer has shared a stateless approach to implementing a 'card of the day' feature that avoids the need for per-user database writes. The method uses a deterministic hash function — such as FNV-1a or xxhash — seeded with a combination of user ID, date, and deck version to always return the same card for a given user on a given day. Timezone handling remains a practical challenge, and the author settled on accepting client-reported dates while validating them within a ±1-day window of UTC. Including a deck version in the seed prevents retroactive card shifts when new cards are added to the deck. The technique is already in use on a live app called Oraclely and is best suited for ephemeral draws where storing draw history is not required.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why Generating Two Recognisable People in One AI Image Remains Unsolved

Creating a convincing AI-generated image of two specific people together remains a significant technical challenge, even as single-subject generation has become largely reliable. When two reference identities are fed into the same generation, models lack an inherent mechanism to keep them distinct, often blending facial features or transferring attributes between subjects. Separate generation and compositing preserves individual likenesses but produces visibly fake results due to mismatched lighting, colour temperature, and perspective. Techniques such as spatial conditioning, per-region identity embeddings, and dedicated handling of contact poses offer partial solutions, but each addresses only one dimension of the problem. A developer working on a tool called DuoPortrait notes that while specific fixes evolve quickly, the core failure modes — identity bleed, anatomical errors at contact points, and lighting inconsistency — have proven stubbornly persistent.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why Japanese and Korean Text Breaks Standard NLP Pipelines

Most text processing tools are built around the assumption that words are separated by spaces, an assumption that Japanese and Korean fundamentally violate. Japanese has no word delimiters and requires morphological analysers like MeCab or Sudachi, which can disagree on how to segment the same text. Korean includes spacing but attaches grammatical particles directly to word stems, meaning a simple split() treats the same noun as multiple unrelated tokens. Unicode normalisation differences in Korean Hangul can also cause the same visible text to have varying code point counts, leading to subtle bugs in length calculations. Accurate processing of both languages requires morphological analysis over raw characters, pinned analyser versions, and validation on natively written text rather than translated content.

0 comments Read more at DEV Community