Transformer

Definition

The neural network architecture behind virtually all modern AI language models. Introduced in the 2017 paper 'Attention Is All You Need,' it replaced recurrent networks by processing entire sequences in parallel using attention mechanisms. Powers GPT, Claude, Gemini, LLaMA, and almost every LLM you've heard of.

How It Works

The transformer is the neural network architecture that powers virtually all modern language models and an increasing share of vision and multimodal models. Introduced in the 2017 paper "Attention Is All You Need" by Google researchers, it replaced recurrent and convolutional approaches with a simpler, parallelizable design based entirely on attention mechanisms.

The key innovation was self-attention: letting every position in a sequence directly attend to every other position. Previous architectures processed tokens sequentially, which was slow and made it hard to capture long-range dependencies. Transformers process all positions simultaneously, making them much faster to train on modern GPU hardware. The architecture also includes feed-forward layers, layer normalization, and residual connections, all organized into repeating blocks that can be stacked deep.

The impact of the transformer is hard to overstate. GPT, Claude, Gemini, LLaMA, BERT, T5, Whisper — they're all transformers. So are modern image models (Vision Transformers) and protein structure predictors (AlphaFold 2). The architecture scales remarkably well: you can make transformers better by making them bigger, and they train efficiently on parallel hardware. Some researchers are exploring alternatives (state-space models like Mamba), but transformers remain the dominant paradigm. The paper that introduced them has over 100,000 citations — it genuinely changed the trajectory of computer science.