Open-source RAG evaluator flags uncertainty instead of guessing blindly

A developer has released rag-triad, a lightweight local evaluator for retrieval-augmented generation (RAG) systems that prioritizes honesty over false confidence. Unlike most AI evaluation tools that assign scores uniformly, rag-triad abstains from scoring when it cannot make a reliable determination, signaling uncertainty explicitly. The tool assesses three distinct failure points in RAG pipelines — poor retrieval, hallucinated output, and off-topic responses — using deterministic checks rather than relying solely on an LLM judge. A key feature called fail-closed groundedness requires the model to cite a verifiable quote from the source context, with code confirming its presence before the check can pass. The project is open-source under the MIT license and runs locally via Ollama, with source code available on GitHub.