Overfitting

Definition

When a model memorizes the training data so well that it performs poorly on new, unseen data. It learns the noise along with the signal. Signs include a big gap between training accuracy (high) and test accuracy (low). Prevented through regularization, dropout, data augmentation, and early stopping.

How It Works

Overfitting happens when a model memorizes its training data instead of learning generalizable patterns. An overfitted model performs brilliantly on data it's seen before but falls apart on anything new. It's like a student who memorizes practice test answers word-for-word but can't handle questions phrased slightly differently.

You can spot overfitting by comparing training performance to validation performance. If training accuracy is 99% but validation accuracy is 60%, the model has memorized rather than learned. Common causes include training for too many epochs, using too little training data, or having a model that's too complex for the task (too many parameters relative to the data).

Preventing overfitting is a core concern in machine learning. Techniques include regularization (adding a penalty for complexity), dropout (randomly deactivating neurons during training), data augmentation (creating variations of training data), and early stopping (halting training when validation performance starts declining). With modern LLMs, overfitting during pre-training is less of a concern because the training data is so vast, but it absolutely matters during fine-tuning where datasets are smaller.