Study Finds AI Judges Miss Omissions When Evaluating Clinical Notes

A new research paper published on arXiv identifies a critical flaw in using large language models as evaluators of AI-generated clinical notes. The study found that LLM-based judges are effective at verifying information that is present in a document but consistently fail to detect missing or omitted information. This 'omission blindness' poses significant risks in medical settings, where absent details — such as drug interactions or patient history — can be clinically dangerous. The findings raise concerns about relying on LLMs for quality assurance in healthcare documentation. Researchers suggest this limitation must be addressed before AI evaluation tools can be safely deployed in clinical workflows.