Data Repetition: The Hidden Factor in AI Training

In the intricate world of AI training, where models grow ever larger and more complex, a new revelation is shaking the foundation of how we mix and use training data. It's not just about scale anymore. It's about repetition rates.

The Repetition Conundrum

As anyone entrenched in AI development knows, pre-training data mixtures are often fine-tuned through small-scale experiments. These proxies are assumed to scale up effectively to larger training budgets. However, when high-quality data is scarce, relying on these small-scale experiments can lead to significant mismatches. Why? Because these datasets are repeated at different rates as the training budget grows, leading to shifts in optimal mixtures that small experiments simply can't predict.

Consider this: In a two-source setup where limited high-quality data is paired with more abundant web crawl data, a single repetition-controlled experiment using just 1/16 of the target tokens can almost perfectly replicate the optimal mixture for a model with 757 million parameters. That's an error margin of just 0.05. Without this control, error jumps to 0.75. The compute layer needs a payment rail that accounts for repetition dynamics, not just scale.

Rethinking Mixture Optimization

What does this mean for AI developers? It suggests a fundamental shift in how we treat data repetition in mixture optimization. Instead of viewing it as an inconvenient side effect of limited data, it's time it becomes a primary variable. The AI-AI Venn diagram is getting thicker, and repetition dynamics are at its core.

When more than two data sources are involved, the complexity increases. Yet, even at the 757M parameter scale, just two repetition-controlled experiments can successfully recover the optimal mixture. Compare this with traditional approaches that demand full-scale experiments, consuming vast resources without guaranteeing the same level of precision.

Why This Matters

Why should this matter to you? If you're in the business of optimizing AI models, understanding and controlling repetition rates can lead to significant cost and time savings. It challenges the notion that scale alone dictates the success of small-scale experiments. Are we underestimating the power of repetition control in our AI pipelines?

In a field where every token counts and efficiency is king, acknowledging the role of repetition could redefine how we approach pre-training. We're building the financial plumbing for machines, and it's time to include repetition as a key component of that infrastructure.