The Language Benchmark Riddle: What's Really Being Measured?

In the relentless pursuit of developing frontier models, multilingual benchmarks have become a guiding star. Yet, despite their popularity, these benchmarks might be missing the point entirely evaluating genuine multilingual proficiency. What they're not telling you: these evaluations often measure mathematical reasoning and factual recall, rather than true language acumen. And that's a problem.

The Benchmark Blind Spot

Let's apply some rigor here. It's become evident that current multilingual benchmarks, much like their monolingual counterparts, are structured to assess mathematical reasoning and factual recall. these are important skills, but they don't necessarily equate to multilingual proficiency. Think about it: if a model excels at arithmetic operations and memorizing facts, does it truly understand the nuances of multiple languages? Color me skeptical.

An interesting observation is the performance disparity between 'thinking' variants and 'instruct' variants in these benchmarks. Thinking variants, designed to be more analytical, often outperform their instruct counterparts on these tests. Yet, real-world multilingual tasks, such as those in LMArena, the tables turn. These models stumble, highlighting a critical gap between benchmark success and practical application.

A New Path: Round-Trip Translation

So, what's the alternative? Enter round-trip translation. This approach involves translating text from a source language to a target language and back again. Any semantic discrepancies that arise during this process expose the shortcomings in a model's multilingual capabilities. Interestingly, this method shows an almost perfect correlation (ρ = 0.94) with user ratings on LMArena, suggesting it's a strong contender for a more realistic evaluation.

What's particularly compelling is that round-trip translation doesn't demand human reference translations or a more sophisticated judge than the model itself. It's a straightforward yet effective way to assess multilingual prowess, bypassing the need for exhaustive manual evaluation that often plagues traditional benchmarks.

Introducing Lost in Translation (LiT)

To push the envelope further, the introduction of Lost in Translation (LiT) offers a rigorous round-trip translation benchmark. LiT spans a diverse array of widely spoken languages, providing a comprehensive platform to evaluate multilingual frontier models realistically. It's a bold step towards aligning benchmarks with the real-world challenges these models are meant to solve.

The question is, will the AI community embrace this shift? Given the entrenched nature of existing benchmarks, change won't come easily. But if we're serious about achieving true multilingual proficiency, it's time to reconsider our evaluation methodologies. After all, isn't the goal to create models that understand and generate languages as naturally as humans do?