Rethinking LLM Post-Training: Beyond the Simplistic Divide

In the ongoing discourse about large language models (LLMs) and their post-training methodologies, a common narrative pits supervised finetuning (SFT) against reinforcement learning (RL). The former is often characterized as mere memorization, while the latter is seen as a gateway to generalization. Yet, this binary view may be oversimplified, particularly reasoning tasks with long chain-of-thought (CoT) supervision.

The Conditional Nature of Generalization

Recent inquiries challenge the notion that SFT lacks cross-domain generalization. The real picture is more complex, hinging on factors such as optimization dynamics, the quality of training data, and the inherent capabilities of the base model. Consider the so-called 'dip-and-recovery' pattern: during training, cross-domain performance might initially decline, only to improve significantly as training progresses. This suggests that short-training checkpoints can misrepresent a model's potential for generalization.

not all data are created equal. Models trained on low-quality solutions tend to falter in generalization. In contrast, those exposed to verified, well-structured long-CoT traces demonstrate consistent improvements across domains. Evidently, the structure and quality of data are as essential as the data itself.

Model Capability: A essential Determinant

The capability of the model can't be overlooked. Stronger models have the advantage of internalizing procedural patterns that transcend domains, even from seemingly trivial tasks like a toy arithmetic game. Weaker models, on the other hand, often mimic surface-level verbosity without deeper understanding. This distinction underscores the importance of selecting the right model architecture for the task at hand.

Interestingly, the generalization effect isn't uniform. While reasoning abilities may improve, the same training can lead to a degradation in safety. This brings us to a turning point question: under what conditions do we prioritize reasoning over safety, and at what cost? This asymmetric generalization reframes the discourse from a simple yes-or-no question to a more nuanced evaluation.

Implications for Future Training Strategies

The implications of these findings are significant. If short-term assessments can mislead our understanding of a model's generalization abilities, should we not rethink our evaluation methods? Perhaps the industry needs to adopt longer training horizons and more rigorous data quality standards to truly gauge a model's potential.

Ultimately, the debate isn't merely about choosing between SFT and RL but understanding the complex interplay of factors that influence generalization. As the field evolves, so too must our strategies and expectations.