Building Crimean Tatar Speech Recognition: Why Measurement Mattered More Than Training

A developer building automatic speech recognition (ASR) for Crimean Tatar, a critically low-resource language, found that fine-tuning a Whisper model on a single consumer GPU took just 90 minutes, but rigorous evaluation took months. Initial scoring revealed a 34.6% word error rate — meaning roughly one in three words was wrong — a problem only discovered after the model had been in use for nearly a year without formal benchmarking. A carefully structured train-test split was designed to avoid data leakage by holding out entire books and unseen readers, rather than using a standard random split. Despite zero filename overlap between training and evaluation sets, a text-based 6-gram check uncovered that four audiobooks had been duplicated under different filenames, silently contaminating prior results. The experience underscores that in low-resource speech projects, data integrity and meaningful evaluation are far harder — and more consequential — than the model training itself.