Rethinking MoE: The Surprising Power of Tiny Dimensions

Mixture-of-Experts (MoE) models have long been celebrated for their remarkable performance. However, the large parameter size has been a persistent thorn in the side deployment. What’s the real snag here? It's not just the size, but the granularity of where critical capabilities are hiding.

Unpacking the Compression Dilemma

Traditional MoE compression methods have stumbled, especially when stretched beyond their typical benchmarks like commonsense reasoning. The issue isn't that these methods don't work at all, but rather that they fail to maintain performance across diverse tasks. The core of the problem lies in the distribution of abilities across the model's architecture.

It turns out that important capabilities aren't evenly spread but are instead concentrated in certain intermediate dimensions of the Feed-Forward Networks (FFNs). This discovery was made using a technique called Fisher importance. Unlike other methods which rely on activation or router scores, Fisher importance has a knack for pinpointing the dimensions that really matter.

Fisher-MoE: A major shift?

Enter Fisher-MoE, the model that seizes on this insight by zeroing in on those important FFN dimensions. In the Qwen1.5-MoE model, merely removing 12 of the 1.35 million routed-FFN dimensions was enough to significantly impact GSM8K accuracy. Yet, it managed to preserve the model’s factual knowledge prowess. This is where the real value lies.

With Fisher-MoE, at a 50% MoE compression ratio, model capability remains largely intact. More than that, it slashes weight memory by around 45% and boosts inference throughput by 21%. These aren't just numbers. They signal a shift in how we might approach model efficiency without compromising on performance.

Why Should This Matter?

So, what does all this mean for the future of MoE models? The implications are clear. If intermediate dimensions hold the key to effective compression and performance retention, then the industry might need to pivot its focus. Why continue to wrestle with unwieldy models when the solution might lie in simply identifying the right dimensions to target?

In a space where speed and efficiency are everything, the ability to maintain performance while reducing size and boosting throughput is a major shift. The real estate industry moves in decades, but technology, much like blockchain, wants to move in blocks. Will the industry embrace this granular approach? The compliance layer is where most of these platforms will live or die.