Mixture of Experts

Definition

An architecture where multiple specialized sub-networks (experts) share a model, but only a few activate for each input. A router decides which experts to use. This lets models be massive in total parameters while keeping compute per token manageable. Mixtral and reportedly GPT-4 use this approach.

How It Works

Mixture of Experts (MoE) is an architecture where a model contains multiple specialized sub-networks ("experts") and a routing mechanism that selects which experts to activate for each input. Instead of running the entire model for every token, only a fraction of the parameters are active at any time. This lets you build models with enormous total parameter counts while keeping compute costs manageable.

GPT-4 is widely believed to be a MoE model, reportedly with 8 experts and about 220 billion parameters each (1.76 trillion total), but only 2 experts activate per token. Mixtral from Mistral AI openly uses MoE with 8 experts, 7B each, activating 2 at a time — so it has 47B total parameters but uses only about 13B per token, making it much faster than a dense 47B model.

The routing decision is where the magic happens. A small gating network looks at each token and decides which experts should handle it. Different experts naturally specialize in different things — one might handle code, another might be better at reasoning, another at languages. The challenge is training the router well and ensuring all experts get used ("load balancing"). When it works, MoE gives you the quality of a huge model at the speed of a much smaller one.