A mechanism that lets neural networks focus on the most relevant parts of their input when producing output.
A mechanism that lets neural networks focus on the most relevant parts of their input when producing output. Instead of treating all input equally, attention assigns different weights to different parts. The 'Attention Is All You Need' paper introduced the transformer architecture built entirely on this idea.
The neural network architecture behind virtually all modern AI language models.
An attention mechanism where a sequence attends to itself — each element looks at all other elements to understand relationships.
An extension of the attention mechanism that runs multiple attention operations in parallel, each with different learned projections.
A mathematical function applied to a neuron's output that introduces non-linearity into the network.
An optimization algorithm that combines the best parts of two other methods — AdaGrad and RMSProp.
Artificial General Intelligence.
Browse our complete glossary or subscribe to our newsletter for the latest AI news and insights.