The Romanization Gap: LLMs Struggle with India's Linguistic Reality

Large Language Models (LLMs) are being thrust into high-stakes roles, particularly in clinical applications across India. Yet, there's a pressing issue: many Indian language speakers prefer to use romanized text over native scripts. This isn't just a quirk, but a systemic challenge. Surprisingly, existing research barely scratches the surface of how this orthographic variation affects real-world applications.

The Romanization Challenge

Let's cut to the chase. Romanization is impacting the reliability of LLMs in essential areas like maternal and newborn healthcare triage. A comprehensive benchmarking of leading LLMs was conducted on a dataset consisting of user-generated health queries across five Indian languages and Nepali. The key finding? A consistent degradation in performance for romanized messages, with the gap reaching up to 24 points across languages and models. That's significant.

Why does this matter? Imagine the implications. At one of the partner maternal health organizations alone, this gap could potentially lead to nearly 2 million excess errors in healthcare triage. It's a staggering number that highlights the critical safety blind spot in LLM-based health systems. Models that seem to understand romanized input may, in fact, fail to act on it reliably.

Bridging the Gap

The paper's key contribution is an innovative approach to address this issue: an Uncertainty-based Selective Routing method. This method's aim is to close the crippling script gap. But there's a broader question here. Shouldn't LLM developers have anticipated this need? In a multilingual world, the ability to handle orthographic variation isn't just nice-to-have. it's essential.

that this builds on prior work from the field, yet the urgency of the problem demands faster solutions. Is the tech community moving swiftly enough to adapt these models for diverse linguistic contexts? If LLMs are to be truly effective in global applications, overcoming the romanization challenge should be a priority.

Looking Forward

This study underscores a vital point. For AI to effectively serve in varied socio-linguistic landscapes, its developers must prioritize regional linguistic nuances. LLMs need to be more than just proficient in English or native scripts. they should be versatile enough to handle the orthographic reality of their users.

Code and data are available at to enable reproducibility and further research. The ablation study reveals critical insights into what needs fixing. But let's not ignore the core issue. Romanization isn't a temporary hurdle, it's a reality for a significant portion of the global population.

, the tech community needs to recognize and address these linguistic gaps with urgency. As LLMs increasingly integrate into high-stakes domains, ensuring their reliability across all script variations isn't just a technical challenge. It's a necessity for global inclusivity and safety.