AI Models Silently Drop Critical Data in Up to 22% of Tasks, Study Finds

A new study tested four AI models — two public open-source models and two compressed versions — on their ability to accurately handle 40 common business identifiers such as tax IDs, card numbers, and patient records. Each model was asked to copy, extract, and summarize these identifiers across 120 checks per model, repeated eight times to establish reliable error rates. One model silently omitted or withheld data in 21.7% of cases, while another failed 3.3% of the time, revealing a sixfold difference in silent data loss between models. Crucially, the failures produced no error messages or refusals — outputs appeared complete and well-formed, meaning standard automated checks would not detect the missing information. The researchers argue that silent data omission should be a primary criterion when selecting AI models for regulated industries like banking, healthcare, and insurance, and that the testing methodology is simple enough for any organization to replicate.