An extension of the attention mechanism that runs multiple attention operations in parallel, each with different learned projections.
An extension of the attention mechanism that runs multiple attention operations in parallel, each with different learned projections. This lets the model attend to different types of relationships simultaneously — one head might focus on syntax, another on semantics, another on position. Core to transformer architecture.
A mechanism that lets neural networks focus on the most relevant parts of their input when producing output.
The neural network architecture behind virtually all modern AI language models.
An attention mechanism where a sequence attends to itself — each element looks at all other elements to understand relationships.
A mathematical function applied to a neuron's output that introduces non-linearity into the network.
An optimization algorithm that combines the best parts of two other methods — AdaGrad and RMSProp.
Artificial General Intelligence.
Browse our complete glossary or subscribe to our newsletter for the latest AI news and insights.