evalmut tool exposes silent failures in AI eval suites using mutation testing

A developer has released evalmut, an open-source tool designed to detect flaws in AI evaluation graders that appear to pass while missing critical errors. The tool works by injecting known semantic defects into test outputs and checking whether the grader still returns a passing verdict, flagging any case where a provably wrong output goes undetected. Unlike code mutation testing tools such as PIT or Stryker, evalmut operates at the grader level and establishes ground truth independently of the grader being tested, avoiding false positives from ambiguous mutations. When run against its own underlying grading engine, gradecore, the tool identified three real holes — one broken check and two coverage gaps — out of 35 mutations applied. The tool is deterministic, requires no LLM-as-judge component, and is available on GitHub via pip install.