Language Models: A Somali Safety Challenge

The safety of large language models, which remain predominantly centered on English, presents a glaring gap low-resource languages like Somali. A recent evaluation of four instruction-tuned models on the SomaliBench v0 has brought this issue into sharper focus. These models, including Llama-3.1-8B-Instruct and Gemma-2-9B-Instruct, were subjected to a hundred harmful-intent prompts paired across both English and Somali.

The English Bias

When models like the Aya-23-8B and Qwen-2.5-7B were run locally with a uniform, English-based 'helpful, harmless, and honest' prompt, the results were telling. The refusal rates, the models' ability to reject harmful prompts, showed a stark contrast between the two languages. For instance, Llama-3.1-8B's refusal rate was an impressive 90% in English but left much to be desired in Somali.

This isn't just a statistical anomaly. It's a reflection of a larger issue: the disproportionate focus on English in AI safety measures. What good is a refusal rate when it's language-specific? Does it not underscore a need for global models that genuinely understand and process languages on an equal footing?

Somali: The Overlooked Frontier

In Somali, the models often failed to produce coherent responses, let alone refuse harmful prompts adequately. Instead of clear refusals, the outputs frequently fell into the category of non-fluent compliance or simply incoherent gibberish. This is more than a technical oversight. It's a fundamental shortcoming in AI deployment, especially as these models are used globally.

The native author's spot-check verification, achieving full agreement with the automated classifications, highlights the reliability of these observations. Yet, it also emphasizes the gap, bridging it's not a matter of tweaking algorithms but rethinking how AI is developed and trained. Is the AI community genuinely committed to inclusivity?

A Call for Inclusive AI

While the aggregate refusal rates and gaps in categories were reported, the raw model generations remain closely held. This is where the enforcement mechanism is where this gets interesting. Without public scrutiny, how can we hold these models accountable for their shortcomings in languages like Somali?

The AI Act text specifies the need for inclusive technological development. However, harmonization sounds clean. The reality is 27 national interpretations, and this diversity is mirrored in the linguistic capabilities, or lack thereof, of AI models. Without a concerted effort toward linguistic inclusivity, AI's promise remains unfulfilled for vast swathes of the globe.