How duplicate chunks and broken caching made local RAG take 291 seconds per query

·1 views

A developer building an offline RAG desktop app on llama.cpp discovered their system was taking nearly five minutes to answer simple questions across a nine-file project. Investigation revealed two root causes: duplicate chunks from mismatched lexical and vector search indexes were sending roughly 9,000 redundant tokens per query, and a misconfigured prompt cache was resetting to near-zero reuse because service calls shared the same slot as conversations. Fixing deduplication and splitting cache slots into separate pools for service calls and conversations pushed cache hit rates from 0% to 45–48%, dramatically cutting prefill costs. The developer also introduced a tiered response strategy: lightweight structured summaries generated once at import for broad questions, and narrow vector retrieval reserved for specific lookups. The key insight was that prefill is not a cost that can be eliminated, only reduced by ensuring the model reads less redundant content per query.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Aegis Latent Core Offers Durable Evidence Trails and MMR Proofs for LLM Traffic

Aegis Latent Core is an AI governance and evidence gateway designed to manage and audit governed large language model (LLM) traffic between applications and model providers. The system enforces admission controls, input canonicalization, and policy checks before forwarding requests to upstream providers. A core feature is its append-only JSONL write-ahead log (WAL), which records request and response hashes, chain linkage, and portable Merkle Mountain Range (MMR) inclusion proofs to create durable, verifiable evidence records. For streaming traffic, sanitized events are emitted incrementally while the success terminal marker is withheld until a final summary is committed to the WAL. The project, currently at version 4.0.0 with a GitHub release labeled v4.0.1, includes Python and TypeScript integrations and an optional Rust-backed auxiliary segment, though the JSONL WAL remains the sole authoritative replay record.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

AI Interviews at 2 A.M. Benefit Employers More Than Candidates, Critics Say

AI-led job interviews conducted outside business hours are being marketed by companies like CodeSignal and Eightfold AI as flexible options for candidates, but critics argue the real beneficiary is the employer. Data shows that one in three candidates on CodeSignal completes AI interviews outside regular hours, while Ribbon AI reports 25% of its bookings fall between 10 p.m. and 2 a.m. Unlike traditional interviews, AI-only formats prevent candidates from asking questions, observing hiring managers, or assessing workplace culture — tools that help job seekers evaluate a potential employer. A survey of nearly 3,000 U.S. job seekers found that 38% have withdrawn from applications upon learning AI would be involved, with 70% saying they were never informed upfront that an AI would evaluate them. Advocates of the format say it improves scheduling flexibility, but critics contend it strips candidates of the reciprocal assessment that defines a fair hiring process.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Calculate How Many Concurrent Builds Your VPS Can Actually Handle

The number of simultaneous builds a VPS can run depends entirely on available RAM, not on the deployment tool being used. A typical Next.js, Nuxt, Remix, or React Router build requires roughly 2 GB of RAM at peak usage, setting a hard floor for any server running such projects. Developers can determine their server's true capacity using three Linux commands that measure free memory, peak build RAM usage, and whether a build survives a set memory ceiling. Running more builds than the available RAM comfortably supports does not immediately cause build failures — instead, it squeezes memory from live applications, leading to 504 errors for site visitors before deploys themselves crash. The safe concurrent build count is calculated as available RAM minus headroom, divided by peak RAM per build, and should be measured directly on the server rather than assumed from general guidelines.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Live Deployment Caught Two Contract Flaws That Passed All Synthetic Tests

A software developer building a release-evidence tool discovered two design defects in the deployment contract only after running it against a real production target, a static dashboard hosted on GitHub Pages. The first flaw assumed every deployment would pass through a preview environment, which GitHub Pages does not provide, forcing a contract revision to allow a direct prepared-to-production transition with an explicit flag. The second defect was a circular dependency, where the release bundle required the finished manifest to compute its digest, and the manifest simultaneously required the finished bundle. Removing the mutual reference made the evidence chain strictly one-directional, resolving the deadlock. The corrected contract was verified against a live production read-back on August 22, 2026, before being frozen, confirming that real-world deployment testing is necessary to surface flaws that synthetic fixtures miss.

0 comments Read more at DEV Community