Why American English Dominates Language Models: A Deep Dive

Large language models (LLMs) are rapidly becoming the backbone of AI applications in high-stakes areas. Yet, they exhibit a distinct bias towards American English (AmE), sidelining other dialects, most notably British English (BrE). This isn't just a linguistic quirk, it's a consequence of geopolitical histories intertwined with data curation and digital dominance.

American English: The De Facto Norm

Our analysis reveals that AmE is overwhelmingly favored in the training of LLMs. A curated examination of 1,813 AmE-BrE variants highlights a systematic tilt towards AmE. But why does this matter? Because the container doesn't care about your consensus mechanism. LLMs are shaping the future of communication, and this bias leads to linguistic homogenization.

The project introduced DiAlign, a novel method to measure alignment between dialects without additional training. By auditing six major pretraining corpora, it was evident that AmE skew isn't just an oversight but a foundational bias. Tokenizer analyses revealed that BrE incurs higher segmentation costs, essentially making it 'expensive' for models to process. Consequently, generative evaluations show a persistent preference for AmE in outputs.

The Cost of Linguistic Bias

Why should we care about dialectal biases in AI? For starters, it raises concerns about epistemic injustice and equity. In a world where AI aims for inclusivity, privileging one dialect over others undermines this goal. Nobody is modelizing lettuce for speculation. They're doing it for traceability and accuracy in representation.

Is this linguistic bias a form of digital colonialism? While it may not be intentional, it echoes historical patterns of dominance. This focus on AmE limits the scope of LLMs, potentially impacting non-AmE speakers' access to fully optimized AI interactions. The ROI isn't in the model. It's in the 40% reduction in document processing time that a more dialectally inclusive model could achieve.

Moving Towards Inclusion

Addressing these biases requires practical steps towards inclusion. As AI technology continues to expand globally, ensuring dialectal diversity isn't merely an academic exercise, it's a business imperative. The global market won't be satisfied with a one-size-fits-all approach. Enterprise AI is boring. That's why it works.

, the dominance of American English in LLMs is more than a mere technical curiosity. It's a call to action for developers, researchers, and policymakers to broaden the horizons of AI. The time has come to embrace linguistic diversity and ensure that AI technologies serve the entire global community.