Decoding MoE: A Deep Dive Into Expert Specialization

Mixture of Experts (MoEs) have become a cornerstone in the evolution of large language models, yet the underlying dynamics of 'expert specialization' remain elusive. The secret sauce appears to be hidden state similarity, not the routing architecture itself. This revelation shifts the focus from the routers to the representation space, where specialization naturally emerges.

Expert Activation Unpacked

The research reveals that MoE routers, essentially linear maps, rely heavily on hidden state similarity to determine expert usage. This mechanism has both token and sequence-level validity across five pre-trained models. Intriguingly, the load-balancing loss plays a important role, suppressing shared hidden state directions to preserve routing diversity. It uncovers why specialization might collapse under less diverse datasets, particularly small batches.

Despite this clarity, MoEs' specialization patterns often baffle human interpreters. Why does expert overlap between different models tackling identical questions hover around only 60%? And why don't prompt-level routing decisions predict the rollout-level ones? These questions highlight the complexities of expert activation.

Challenges in Interpretation

The deeper layers of MoEs seem almost defiant in their consistency. They show near-identical expert activation across semantically unrelated inputs, especially in reasoning models. This phenomenon begs the question: How much do we really understand about MoEs? If anything, it highlights a critical gap in our comprehension of LLM hidden state geometry, a challenge that has long puzzled the field.

So, what's the takeaway? The efficiency of MoEs in processing language is well-documented. However, cracking the code on expert specialization remains a tantalizing mystery. It’s not just about efficiency. it’s about unlocking a deeper understanding of how these models think. In a way, the AI-AI Venn diagram is getting thicker, and we’re only scratching the surface of its potential.