Autoregressive Model

Definition

A model that generates output one piece at a time, with each new piece depending on all the previous ones. GPT and other large language models work this way — they predict the next token based on everything that came before it. Great for text generation, but inherently sequential.

How It Works

Autoregressive models generate output one piece at a time, where each new piece depends on everything that came before it. In language models, this means predicting one token, appending it to the sequence, then predicting the next token, and so on. It's like writing a sentence where each word choice constrains what comes next.

This approach is what makes chatbots feel like they're "thinking" as they type — they literally are generating one token at a time. GPT, Claude, LLaMA, and virtually every modern language model works this way. The tradeoff is speed: since each token depends on the previous ones, you can't easily parallelize generation. That's why responses take time to stream in.

The alternative approaches — like masked language models (BERT) or diffusion models — work differently. BERT predicts missing words in the middle of sentences (good for understanding, not great for generation). Diffusion models start with noise and refine it into output. But for open-ended text generation, autoregressive models still dominate because they naturally capture the left-to-right flow of language.