The initial, expensive phase of training where a model learns general patterns from a massive dataset.
The initial, expensive phase of training where a model learns general patterns from a massive dataset. For language models, this typically means next-token prediction on trillions of tokens scraped from the internet. Pre-training gives the model broad knowledge that fine-tuning later specializes.
The process of taking a pre-trained model and continuing to train it on a smaller, specific dataset to adapt it for a particular task or domain.
A large AI model trained on broad data that can be adapted for many different tasks.
The process of teaching an AI model by exposing it to data and adjusting its parameters to minimize errors.
A mathematical function applied to a neuron's output that introduces non-linearity into the network.
An optimization algorithm that combines the best parts of two other methods — AdaGrad and RMSProp.
Artificial General Intelligence.
Browse our complete glossary or subscribe to our newsletter for the latest AI news and insights.