An architecture where multiple specialized sub-networks (experts) share a model, but only a few activate for each input.
An architecture where multiple specialized sub-networks (experts) share a model, but only a few activate for each input. A router decides which experts to use. This lets models be massive in total parameters while keeping compute per token manageable. Mixtral and reportedly GPT-4 use this approach.
The neural network architecture behind virtually all modern AI language models.
A mathematical function applied to a neuron's output that introduces non-linearity into the network.
An optimization algorithm that combines the best parts of two other methods — AdaGrad and RMSProp.
Artificial General Intelligence.
The research field focused on making sure AI systems do what humans actually want them to do.
The broad field studying how to build AI systems that are safe, reliable, and beneficial.
Browse our complete glossary or subscribe to our newsletter for the latest AI news and insights.