muteval Tool Detects Missing LLM Tests by Deliberately Degrading System Prompts
A developer has released an open-source tool called muteval that identifies gaps in LLM evaluation suites by intentionally degrading system prompts and observing whether existing tests catch the change. Unlike standard test runners that report failed assertions, muteval flags cases where the model's output changed but all checks still passed — called 'survivors' — pointing to untested behaviors. The tool works offline without an API key, using a deterministic mock model to isolate suite blind spots from model variability. A demo shows how deleting a single prompt rule, such as 'reply in English,' goes undetected by a passing test suite that never checked for language compliance. The tool is scoped to mutations it can perform, such as dropping lines or corrupting retrieved documents, and surfaces coverage gaps rather than rendering verdicts on system quality.
