Romanized Nepali: Cracking the Code with LLMs

Romanized Nepali, the script used for Nepali in the Latin alphabet, continues to be a significant blind spot for large language models (LLMs). Yet, its relevance in digital communication across Nepal demands attention. In a recent study, researchers tackled this underrepresented challenge using three open-weight models: Llama-3.1-8B, Mistral-7B-v0.1, and Qwen3-8B. The findings are a mixed bag of breakthroughs and enduring hurdles.

Benchmarking the Models

With a dataset of 10,000 Romanized Nepali phrases, the models were put to the test in both zero-shot and fine-tuned scenarios. The evaluation employed metrics like Perplexity (PPL), BERTScore, and BLEU to measure fluency and semantic integrity. Initially, none of the models could generate Romanized Nepali correctly without fine-tuning. Each stumbled over its unique architectural quirks. But post fine-tuning, they showed promise. Qwen3-8B, in particular, emerged as a frontrunner, becoming the only model to deliver semantically relevant outputs in a zero-shot setting. That's noteworthy, yet it's just half the story.

Fine-Tuning: A Double-Edged Sword?

Llama-3.1-8B had the weakest zero-shot performance but gained significantly after fine-tuning, boasting a Perplexity drop of 49.77 points. This raises a critical question: Are we too dependent on fine-tuning for real-world applications? Slapping a model on a GPU rental isn't a convergence thesis. Fine-tuning may fix the immediate issues, but can these models adapt dynamically in the wild? If the AI can hold a wallet, who writes the risk model?

The Path Forward

Romanized Nepali adaptation is set to become a proving ground for LLMs tackling low-resource languages. These findings provide a rigorous baseline for future endeavors. However, the larger question remains: Is this just another academic exercise, or can it pave the way for practical, scalable solutions? Show me the inference costs. Then we'll talk. The intersection is real, but ninety percent of the projects aren't. Until these models can perform consistently in real-time applications, skepticism remains warranted.