Why Direct Translations Won't Cut It for LLM Safety

Trusting a language model with just direct translations? You're likely missing the mark. Multilingual safety evaluation for large language models (LLMs) has leaned heavily on translating English benchmarks. But new research shows this isn't enough. It ignores the nuances of cultural contexts, threat scenarios, and social norms that are important for accurate assessments.

Translating Isn't Enough

Researchers have paired direct translation (DT) and culturally-adapted (CA) datasets for four languages: Korean, Japanese, Thai, and Khmer. They compared Attack Success Rate (ASR) and Cultural Realism scores across four open-source LLMs. The results were clear. CA prompts showed higher success rates, with a Delta-ASR increase of over 9 percentage points on average across all 16 language-model combinations. Simply put, relying on direct translation is underestimating risks in 44 out of 48 category-language combinations.

Why Cultural Context Matters

Language isn't just words. It's context. The distribution of threat forms varies across languages, affecting how LLMs should be evaluated. Cultural Realism scores from DT assessments consistently scored below 1.0 out of 3.0, while CA scores soared up to 2.51. If you're only translating, you're missing the reality of multicultural settings. The stakes are high. If your LLM doesn't understand cultural nuances, how reliable is it really?

The Future of LLM Evaluation

This study shows adapting benchmarks to specific cultural contexts is necessary. And let's be honest, isn't it time we stopped pretending language is one-dimensional? Solana doesn't wait for permission, and neither should our language models. If you haven't considered cultural adaptation yet, you're already late. This is the direction multilingual evaluation needs to go, and fast. The world is a diverse place. Our tech needs to reflect that.