An attention mechanism where a sequence attends to itself — each element looks at all other elements to understand relationships.
An attention mechanism where a sequence attends to itself — each element looks at all other elements to understand relationships. This is what lets transformers capture long-range dependencies in text. The query, key, and value matrices compute attention scores that determine how much each word influences every other word.
A mechanism that lets neural networks focus on the most relevant parts of their input when producing output.
The neural network architecture behind virtually all modern AI language models.
An extension of the attention mechanism that runs multiple attention operations in parallel, each with different learned projections.
A mathematical function applied to a neuron's output that introduces non-linearity into the network.
An optimization algorithm that combines the best parts of two other methods — AdaGrad and RMSProp.
Artificial General Intelligence.
Browse our complete glossary or subscribe to our newsletter for the latest AI news and insights.