Oryx: A New Hybrid Model Changing the Token Game

In the sprawling world of large language models, Softmax attention has long held court. Yet, its memory consumption grows linearly while compute demands rise quadratically with the sequence length. Enter Oryx, a hybrid model that could change the very rules of the game, blending quadratic attention with linear recurrent models to achieve a powerful balance of efficiency and performance.

The Rise of Linear Recurrent Models

Linear recurrent models, like linear attention and state space models, have gained traction as alternatives to traditional attention due to their linear compute and constant memory requirements. These models, or mixers, promise efficiency gains and impressive results across numerous benchmarks. However, they often falter in tasks that demand long-context retrieval or in-context learning, a important aspect where traditional attention still shines.

That's where Oryx steps in, introducing a novel approach by allowing dynamic interaction across the token sequence. Through a flexible architecture, Oryx can switch between various mixers, employing quadratic attention for scenarios requiring rich context and linear recurrences for efficient generation. This adaptability ensures that Oryx can maintain high performance without the usual trade-offs.

Shared Wisdom: Internal Representations

Oryx's strength lies in its ability to share at least 90% of its parameters across different mixers, allowing attention and recurrent modes to operate over shared internal representations. This shared wisdom not only enhances performance but also showcases the potential for attention and linear recurrent models to coexist harmoniously within a single framework.

Comparing Oryx against its single-mixer counterparts presents a compelling picture. At the scale of 1.4 billion parameters, Oryx outperforms these baselines by a margin of at least 0.7 percentage points on average language modeling tasks. A notable feat, achieved with a mixed-training strategy and fixed token budgets, further underscores the model's efficacy.

Implications for the Future

On retrieval tasks, the results are equally impressive. Oryx matches the performance of the Transformer baseline while processing less than 10% of the tokens in attention mode. This efficiency points to a future where large models can be both powerful and manageable, a balance many in the field are eager to achieve.

But let's apply some rigor here. While Oryx shows promise, it's worth questioning whether this hybridization approach will consistently outperform as models scale even further. Can this balance be maintained as we push the boundaries of what's possible in language modeling?

Nonetheless, the introduction of Oryx suggests a promising direction for the development of language models. By enabling attention and linear recurrent models to share internal representations, Oryx not only broadens the horizon for hybrid models but also challenges the industry to rethink the limitations of existing architectures.