Data Repetition: The Hidden Variable in AI Model Training

Pre-training AI models isn't just about throwing data and compute at the problem anymore. It's about precision, particularly when high-quality datasets are limited. A critical oversight in the current approach is the repetition mismatch of these datasets, which shifts the optimal mixture as training budgets grow.

Mismatch in Repetition

When high-quality data is scarce, its repetition rate changes as the training budget scales. This often leads to suboptimal outcomes that small-scale experiments don't predict. A recent study highlights that controlling this repetition variable with a subsampling procedure can significantly enhance the effectiveness of data mixtures.

Consider this: a 757M parameter model could achieve near-optimal data mixtures using just 1/16 of the target tokens if repetition is managed properly. In contrast, ignoring repetition control inflates errors to 0.75, demanding three to four full training horizons. That's up to 94% of the token budget wasted.

The Oversight of High-Quality Data

Incorporating more data sources only complicates this issue. A larger mixture space necessitates more experiments, yet repetition-controlled horizons still outperform traditional methods. At the 757M scale, two well-managed experiments can achieve optimal mixtures, while others flounder with full-scale trials. Why are we so fixated on scale when repetition dynamics hold the key?

Implications for AI Training

Repetition isn't just an inconvenient byproduct of limited data. It's a first-class variable that needs attention in data mixture optimization. AI researchers and engineers need to shift focus from brute force data scaling to managing these nuanced repetition dynamics.

Slapping a model on a GPU rental isn't a convergence thesis. If we can't control basic elements like data repetition, the promise of AI remains stunted. Are we ready to prioritize this hidden variable and refine our approach to pre-training AI models?