Revisiting Old Roads: The LLM Training Paradigm at a Crossroads

The field of large language models (LLMs) finds itself at an intriguing juncture. Recent methods for training these models are strikingly reminiscent of strategies from the BERT age, emphasizing an extensive post-training phase that includes supervised fine-tuning (SFT) and reinforcement learning (RL). This observation, while not entirely new, urges us to reconsider the directions in which these methodologies are heading.

Echoes from the Past

Color me skeptical, but the current training methods for LLMs seem like a trip down memory lane. The methodology mirrors the 'pre-train then fine-tune' approach of earlier times, where models were explicitly adjusted for specific tasks and benchmarks. The resurgence of this strategy suggests that the supposed evolution in LLM training might not be as groundbreaking as some claim. Let's apply some rigor here. Is revisiting old methodologies genuinely the best path forward?

To put this into perspective, a historical overview of LLMs shows phases where task performance heavily depended on fitting models to in-distribution datasets. This is precisely what we're seeing today, albeit with fancier terminologies and more complex datasets. When pre-trained models were compared to randomly initialized ones on modern reasoning datasets, the results struck a familiar chord. Post-trained models from scratch exhibited commendable performance, proving that the emperor might not have as many new clothes as we thought.

The Distribution-Fitting Conundrum

The findings suggest that today’s post-training methodologies primarily function as a distribution-fitting mechanism. This raises the question: are we truly cultivating intelligence in these models, or are we just tuning them to excel in specific niches? The claim doesn't survive scrutiny if it posits that such methods foster genuine generalization.

What they're not telling you: the current approach risks overfitting models to the benchmarks we care about today, at the expense of broader applicability. While it’s admittedly impressive to see models excel at predefined tasks, the broader question of whether they can adapt to unforeseen challenges remains unaddressed. Are these models learning to perform, or merely learning to conform?

A Call for a Paradigm Shift

The path forward, some argue, lies in developing models that 'learn how to learn,' moving beyond the confines of extensive post-training tailored to specific behaviors. This isn't just a technical challenge but a conceptual shift that demands more than just incremental improvements. It requires rethinking the very nature of what it means for a model to learn.

In this context, the task of creating generally capable models that can adapt and thrive in dynamic environments assumes important importance. The stakes are high, and the journey is fraught with both promise and peril. But if the ultimate goal is truly intelligent systems, then a mere rehash of the past won’t suffice. We need to look beyond the tried-and-true and embrace bolder, more innovative approaches.