Tiny 125M model outperforms 14B LLM at medical text de-identification, runs on CPU

A developer built an open-source tool called localscrub designed to remove protected health information (PHI) from clinical text entirely on local hardware, without sending data to cloud services. The system uses a cascade approach, combining regex-based rules, a 125-million-parameter NER model (obi/deid_roberta_i2b2), and a 14-billion-parameter LLM (Qwen3 via Ollama) as fallback stages. Benchmarking on synthetic clinical notes showed the smaller NER model achieved a redaction recall of 0.94 at 184 milliseconds per note on CPU, outperforming the larger LLM in both speed and accuracy on that corpus. The developer evaluated results using three metrics — relaxed recall, strict recall, and redaction recall — with the latter treated as the primary safety measure since it checks whether every character of sensitive data was removed. The project also surfaced several integration bugs and gold-standard labeling errors, highlighting the importance of rigorous, reproducible evaluation when building de-identification pipelines.