Why Arabizi — Arabic Written in Latin Letters and Digits — Trips Up AI Models
Arabizi is a widely used writing system in which Arabic speakers substitute digits for consonants that have no Latin equivalent, such as '7' for ح and '3' for ع, and has been common across the Levant, Egypt, and the Gulf long before smartphones. The core problem lies not in AI models themselves but in their tokenizers, which were trained on corpora containing virtually no Arabizi and therefore split letter-digit boundaries into fragments, turning a single common word like 'el7amdulillah' into six disconnected pieces. This fragmentation inflates token counts and strips words of their meaning, leaving the model to interpret stray digit tokens through a numeric lens drawn from invoices and dates in training data. That numeric bias creates a practical extraction bug: automated pipelines scanning for numbers in a message can mistakenly harvest letter-digits alongside real figures, causing errors in fields like order quantities or reference codes. When models are asked to generate Arabizi in response, they lack consistent training signal for regional conventions and produce inconsistent or garbled transliterations.
