Attention

Definition

A mechanism that lets neural networks focus on the most relevant parts of their input when producing output. Instead of treating all input equally, attention assigns different weights to different parts. The 'Attention Is All You Need' paper introduced the transformer architecture built entirely on this idea.

How It Works

Attention is the mechanism that lets neural networks figure out which parts of their input matter most for any given task. Before attention, models processed sequences step by step and struggled to connect information that was far apart. A model reading a long paragraph might forget what the first sentence said by the time it reached the last one.

The key insight is deceptively simple: instead of forcing information through a bottleneck, let the model look back at all previous positions and decide what's relevant. Each word (or token) gets to "attend" to every other word and assign it an importance weight. The phrase "The cat sat on the mat because it was tired" — attention helps the model figure out that "it" refers to "the cat," not "the mat."

Self-attention, the variant used in transformers, computes three vectors for each token: a query, a key, and a value. The query asks "what am I looking for?", keys say "here's what I contain," and values carry the actual information. Matching queries to keys produces attention weights that determine how much each value contributes. This simple mechanism, scaled up massively, powers every modern language model.