PrismShine tool aims to catch AI hallucinations at the source, not just the output

·1 views

A developer has released PrismShine (version 0.2.2, Apache-2.0), an open-source Python library designed to verify AI-generated answers against the evidence provided to the model at runtime. Unlike most anti-hallucination tools that only score the final text output, PrismShine attempts to identify upstream failure causes such as empty retrieval, tool errors, or stale cache before or after the model generates a response. The tool produces a structured verdict that includes a decision, a named resolution gate, and an evidence hash intended for audit logging. In published benchmarks against Vectara's HHEM-2.1-Open on the HaluEval dataset, PrismShine-fast reportedly achieved a higher QA F1 score of 0.831 versus 0.746, with faster median response times. The library is available via pip and includes a browser-based demo that requires no API key.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Open-Source Tool Maps 100K+ Displaced Museum Artifacts Back to Their Origins

A designer and developer has built Ex Situ, an open-source spatial index that maps over 100,000 museum artifacts held in Western institutions back to their geographic origins using interactive arcs. The project, which began as an MA thesis in 2022, currently indexes collections from institutions including the Met, V&A, and SMB Berlin, covering categories such as Islamic, Asian, African, and ethnological art. Built with Next.js, Deck.gl, and a Strapi backend, it pulls data from museum open-access APIs and is licensed under AGPL-3.0. Rather than duplicating institutional data, the tool links origin sites to destination collections, routing researchers directly to source institutions. The solo developer is now seeking community contributions to help with performance scaling, data pipelines, and missing museum API integrations.

0 comments Read more at Hacker News

ProgrammingDEV Community ·

How to Safely Migrate Large-Scale Metadata Systems in Distributed Environments

Migrating metadata at cloud scale is far more complex than a simple copy-and-cutover process, as the migration itself becomes a distributed system with multiple failure risks. The four primary dangers are data drift during copying, partial record transfers, stale write replays, and premature traffic cutover before validation is complete. A reliable migration follows a structured pipeline: snapshot, copy, replay, validate, cutover, cleanup, and recover, with the source remaining authoritative until the destination is fully verified. To prevent duplicate or out-of-order writes, each destination partition should use monotonically increasing sequence numbers to reject stale or repeated operations. Every stage of the migration must be designed to be retry-safe, ensuring that crashes, partial failures, or delayed writes do not leave the system in an inconsistent state.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Engineering Team Cuts RAG Pipeline Latency 40% Using Bayesian Search and Hybrid Retrieval

A development team overhauled their Retrieval-Augmented Generation (RAG) pipeline after standard production deployments revealed critical failures across legal, API, and support document types. The team replaced fixed 512-token chunking with document-specific strategies — including recursive, semantic, and agentic chunking — achieving recall@10 scores between 91% and 97% depending on content type. They combined vector search with BM25 and cross-encoder reranking using Reciprocal Rank Fusion, reducing reliance on any single retrieval method and improving relevance correlation from roughly 0.75 to 0.92. A query transformation layer was also added to handle poorly formed user queries before retrieval begins. The cumulative changes resulted in a 40% reduction in end-to-end query latency compared to the original pipeline.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How One Team Built Automated LLM Evaluation That Catches 92% of Hallucinations

A software team discovered critical gaps in their AI quality process after deploying a retrieval-augmented generation customer support assistant that served hallucinated responses to over 500 users. The failures included fabricated billing policies and rate-limit figures pulled from a competitor's documentation, exposing the risks of relying solely on informal manual testing. In response, the team built a production-grade evaluation pipeline that runs automatically within CI/CD workflows, using a ensemble of judges to assess faithfulness, instruction-following, schema validity, and safety. The system uses domain-specific LLM judges alongside deterministic checks, with configurable pass thresholds and a versioned golden dataset for regression detection. The approach replaced ad-hoc review with structured, automated metrics capable of blocking low-quality model updates before they reach production.

0 comments Read more at DEV Community