SShortSingh.
Back to feed

Semantic Cache Hit Rates Can Hide Wrong Answers, Seven Tests Show

0
·2 views

A semantic cache reuses stored LLM responses when a new query is similar enough to a past one, and its hit rate alone does not show whether reused answers actually fit the new question. In a test using Actian VectorAI DB at a 0.80 similarity threshold, the cache hit 79.2% of lookups, but 6.4% of those hits returned wrong answers, some with similarity scores above 0.95. The author ran seven tests covering threshold calibration, negation, changed numbers and dates, tenant and intent boundaries, local verification, query length, and stale cached answers. Each test lowered the false-positive rate, though often at the cost of hit rate, and the results are presented as controlled measurements rather than production guarantees. The experiments used 3,075 Banking77 questions as the cache and 770 held-out queries for lookups, with a 5% false-positive target.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

QA contractor's questions about testing Android apps on cloud phones

An operator of DuoPlus, a cloud phone service, shared notes from a month-long engagement with a freelance QA tester testing a client's Android app. The tester was initially skeptical of cloud devices but was drawn to their real ARM hardware and isolated environments, which mattered because the app's native library behaved differently on emulators. The team ran one device per Android version (10, 11, 12 and 15) and used cloud numbers to receive SMS verification codes during signup testing. When the contract ended, the devices were transferred to another tester through team sharing without relying on any personal handset. The operator noted limitations, including Android-only support and per-minute billing that requires cost monitoring, and that the positive outcome was not solely due to the tooling.

0
ProgrammingDEV Community ·

Pune developer builds offline-first Flutter app to help home-visit physiotherapist track visits and fees

A developer created CareLog, a free app for tracking visits and payments, after his physiotherapist wife struggled to keep records for her home-visit patients in Pune. The app stores all patient data only on the phone, which protects privacy and keeps it working in buildings with poor network coverage. Core features include one-tap visit logging, per-patient fee and schedule tracking, overdue dues flagging, and PDF bills with a UPI QR code. Backups are handled by daily automatic on-device copies and user-shared JSON files, and the app supports English, Hindi and Marathi. It is now available on Google Play and as a web app.

0
ProgrammingDEV Community ·

JARVIS Desktop Assistant Ships Bundled Python So Non-Developers Can Install It

The developer of JARVIS, a Python desktop assistant, set a goal that someone with no GitHub or Python experience could download, install, and run the app without using a terminal. To achieve this, the build bundles a relocatable CPython runtime from python-build-standalone for Windows and macOS, verified against a SHA-256 checksum pinned in the repository. The installer unpacks this runtime beside the app rather than system-wide, and swaps in new versions atomically to avoid half-replaced interpreters. A bug was also addressed where the installer would stop the in-app updater mid-update, which was fixed by having it skip its own ancestor processes. Code signing remains unresolved, meaning macOS Gatekeeper and Windows SmartScreen will still show a one-time warning that the download page explains how to bypass.

0
ProgrammingDEV Community ·

Adding Dublin Core Metadata to a Static HTML Page Using dublin-core-basics

A tutorial walks through adding a small, explicit Dublin Core metadata profile to a static HTML page instead of copying long lists of meta tags. It uses the open-source dublin-core-basics repository, an MIT-licensed static guide, as a readable reference for the vocabulary. The author stresses that Dublin Core does not guarantee better search rankings, so the aim is to verify what the browser actually receives. Readers are told to run the repository locally with Python's HTTP server and to check current DCMI specifications before reusing claims or historical values. The piece also notes that metadata describes a resource rather than replacing accurate page content, headings, or a canonical URL.