Unveiling BlasBench: A Leap for Irish ASR Systems

In the maze of automated speech recognition (ASR) technology, Irish-specific systems have long lacked a standardized benchmark. Enter BlasBench, an innovative evaluation platform tailored for Irish nuances. It introduces Irish-aware text normalization, crucially preserving linguistic features like fadas and lenition.

Benchmark Breakdown

BlasBench dives into the performance of 12 ASR systems spread across four major architecture families. It uses datasets from Common Voice ga-IE and FLEURS ga-IE. The Whisper variants, however, fall short, exceeding a 100% Word Error Rate (WER). On the other hand, the omniASR LLM 7B model emerges as the standout performer, achieving a 30.65% WER on Common Voice and 39.09% on FLEURS.

Generalisation Gap Exposed

Here's where it gets interesting. Models fine-tuned on the Common Voice dataset show a significant drop when tested on FLEURS. They lose between 33 to 43 WER points, signaling a glaring generalisation gap that's masked in single-dataset evaluations. The trend is clearer when you see it. The ability of an ASR system to generalize across datasets is critical for real-world applications.

Why This Matters

So, why should we care? Speech recognition systems are marching towards dominance in tech interfaces. Yet, their proficiency in less common languages remains restricted. BlasBench sets a new stage for Irish ASR, ensuring more strong evaluations and pushing developers to address these generalisation gaps.

One chart, one takeaway: The disparity in WER across datasets is a wake-up call. How can we expect ASR systems to serve global populations if they can't handle regional language subtleties? This isn't just about Irish. It's a microcosm for minority languages worldwide.

As the tech community rallies to improve these models, BlasBench offers a measurable way forward. It's not just a technical exercise, it's about equitable access to tech for everyone, regardless of language.