GaelEval's Bold Leap: Gaelic Language Models Surpass Human Benchmarks

Language models are evolving rapidly, and the latest benchmark, GaelEval, offers a fascinating glimpse into their capabilities in minority languages. With a focus on Gaelic, a morphosyntactically rich language, GaelEval puts 19 multilingual large language models (LLMs) through their paces. The results? They're not just good, they're outshining human performance in certain areas.

Gaelic's New Frontier

The benchmark itself is groundbreaking. GaelEval isn't just another language test. It comprises three components: an expertly crafted morphosyntactic multiple-choice questionnaire, a culturally nuanced translation task, and a large-scale Q&A focused on Gaelic cultural knowledge. Each of these tests pushes the models to demonstrate structural and cultural competence in ways previous benchmarks have missed.

And the models have delivered. The Gemini 3 Pro Preview, for instance, scored 83.3% accuracy on the linguistic task, surpassing the human baseline of 78.1%. It's a clear sign that proprietary models, with their closed development environments, have the edge over open-weight systems.

Proprietary Models Take the Lead

This brings us to an intriguing observation: proprietary models consistently outperform their open-weight counterparts. It's a trend that's hard to ignore and raises a critical question, are we seeing the end of the era where open-source systems can compete on equal footing with proprietary giants?

Gaelic prompts also play a role, albeit a smaller one. In-language prompting gave models a slight edge, boosting performance by 2.4%, proving that language context matters even in the area of AI.

Cultural Competence and the Path Forward

On the cultural front, the top models soared past the 90% accuracy mark. Yet, when prompted in Gaelic, most systems stumbled, and their scores appeared inflated compared to manual benchmarks. This suggests that while models excel in certain structured tasks, there's still room for growth in real-world language use.

The court's reasoning hinges on the idea that these models can achieve above-human performance in specific areas of Gaelic grammar. But here's what the ruling actually means: GaelEval doesn't just help us measure LLM progress, it also highlights the gaps. The test is revealing the strengths and weaknesses of current AI models, particularly the divide between proprietary and open-weight systems.

So, where do we go from here? As language models become more adept at handling minority languages like Gaelic, the focus should shift to ensuring these advancements contribute positively to language preservation and cultural understanding, rather than just serving as tech triumphs.