Multilingual LLM Testing Reveals Gaps That English-Only Evals Miss

A developer who evaluated large language model outputs across English, Hindi, Tamil, and Marathi found that model performance varies by the combination of language and task, not language alone. One key finding was that fluent-sounding output in less-common languages is more likely to be accepted uncritically, even when factually wrong. Script errors, diacritic drops, and mixed-language sentences exposed failure modes that standard automated checks and monolingual reviewers routinely overlook. The evaluation also highlighted how code-switching — speakers blending English words into regional language sentences — can disrupt grammar or cause models to mistranslate terms that should stay in English. The core takeaway is that testing a model in one language reveals little about its behaviour in others, making single-language evaluation insufficient for products intended for multilingual audiences.