Trilobyte: A New Wave in Audio Compression

Autoregressive language models are often associated with text, but their potential extends far beyond words. In a groundbreaking twist, these models are being adapted for audio compression, a domain traditionally dominated by codecs like FLAC. The catch? These innovations have mostly hovered around 8-bit audio, leaving many wondering if such models can handle higher bit depths.

Why Bit Depths Matter

Here's what the benchmarks actually show: audio, bit depth is essential. Higher bit depths mean richer, more detailed sound. Yet, they also mean larger vocabulary sizes for tokenization. At 16-bit, we're talking a 65K vocabulary size, which balloons to 16.7 million at 24-bit. That's where the trouble starts.

Enter Trilobyte. This byte-level tokenization schema transforms the complexity of scaling vocabulary size from an exponential nightmare to something much more manageable. With Trilobyte, we're seeing the first practical approach to 24-bit LM-based lossless compression. Frankly, this is a big deal.

The Performance Puzzle

So, how does LM-based compression stack up? At 8-bit and 16-bit, these models outperform FLAC, a codec many consider the gold standard. But strip away the marketing, and you get a more nuanced picture. As bit depths rise, the compression gains become less impressive. This suggests that while language models are promising, they aren't a panacea.

Let's be clear: the numbers tell a different story for higher bit depths. The gains become marginal. Why should readers care? Because the industry is on the cusp of a transformation, and understanding these limitations is key to pushing forward.

Why It Matters

The architecture matters more than the parameter count. That might sound like a bold claim, but the reality is clear. Trilobyte's ability to manage vocabulary size elegantly at higher bit depths could pave the way for future breakthroughs. It's not just about compressing audio. It's about doing it in a way that's scalable and efficient.

Will Trilobyte make other codecs obsolete? Not yet. But its introduction refocuses the conversation on scaling efficiency, which is a step in the right direction. The question isn't if LM-based compression will dominate but how soon it will become the norm.