SiPaKosa: A Game Changer for Buddhist Texts in AI

JUST IN: There's a new player in the language model training ground. SiPaKosa, a massive corpus of Sinhala and Pali doctrinal texts, is making waves. We're talking about 786,000 sentences and a whopping 9.25 million words. That's huge!

What's Inside?

The corpus isn't just any collection. It includes 16 historical Buddhist documents, cleared of copyright, plus the full Tripitaka canonical texts scraped from the web. Google Document AI was used for high-quality OCR on historical manuscripts. That's some serious tech for some seriously old texts.

This treasure trove is neatly divided into language-specific groups: Sinhala and a mixed Sinhala-Pali section. The attention to detail here's wild, with rigorous quality control and metadata annotation to boot.

The Performance Race

Here's where things get spicy. The folks behind this corpus tested ten pretrained models, checking out perplexity scores. These ranged from a super low 1.09 to a far less impressive 189.67. What's clear? Proprietary models are knocking it out of the park, outperforming open-source ones by three to six times. The labs are scrambling to catch up.

And just like that, the leaderboard shifts. Why does it matter? Well, this could change how we train domain-adapted language models, improve historical language analysis, and boost information retrieval systems in Buddhist studies. It's not just about the tech. it's about preserving Sinhala cultural heritage too.

Why Should You Care?

So, why should you care? Simple. This corpus isn't just a fancy collection for scholars to admire. It's a tool for innovation in AI language models. If proprietary models are doing better, what does that mean for open-source development? Is this a sign of things to come in AI research? It might be time to rethink strategies.

Could this also be the push needed to preserve other cultural heritages through AI? If this corpus can do so much for Sinhala and Pali texts, imagine what AI could achieve for other languages and cultures on the brink of being forgotten.