Open-source tool 'laya-evals' adds confidence audits and CI gates for LLM evaluations

Developer NandhaKishorM has released laya-evals, an open-source tool designed to improve LLM evaluation in production systems. The tool audits a model's calibration, measuring whether its stated confidence aligns with actual accuracy, rather than just judging correctness. It provides specific confidence thresholds for different question types and integrates into CI/CD pipelines, failing builds if accuracy or calibration regresses. The tool runs locally at no cost and is the fifth component in the laya decision engine series, which also includes tools for routing, token compression, and phishing detection.