The Subtle Dance of Warm-Start Techniques in Vision-Language Models

Vision-language models (VLMs) have come a long way, and the technique of employing a two-stage post-training process is gaining traction. Stage-1, a warm-start with supervised fine-tuning or on-policy distillation, is followed by Stage-2, where reinforcement learning (RL) takes the helm. But what exactly does Stage-1 change, and why does it matter?

Understanding the Warm-Start

In a study involving the Qwen2.5-VL-7B model, researchers investigated the effects of using a same-modality 72B VLM teacher for on-policy distillation (OPD) during Stage-1. Interestingly, the findings revealed that the different warm-start strategies, whether supervised fine-tuning or OPD, resulted in a narrow performance band of 53-54% on the Geometry3K internal validation. This suggests that Stage-1 may not significantly alter the in-domain endpoint.

Yet, out-of-domain tasks such as MathVista, an early-stopped supervised fine-tuning improved performance by 2.1 points. In contrast, an over-trained variant saw a dramatic 9.5-point drop. This highlights the delicate balance required in this initial stage and raises an important question: Are we focusing enough on optimizing these early steps?

The Entropy Factor

One of the most striking revelations was the difference in entropy regimes between OPD and supervised fine-tuning. OPD entered the RL stage with much higher policy entropy, a distinction that persisted throughout the learning trajectories. This wasn't just an academic curiosity, it manifested in higher answer diversity and improved pass@16 metrics at the in-domain initialization, although the advantage wasn't clear-cut.

However, after reinforcement learning, these distinctions faded. The endpoint pass@16 values were within a mere 1.1 points of each other, and on MathVista, the margin was even thinner at 1.2 points. This implies that while Stage-1 sets the stage, the second act of reinforcement learning levels the playing field.

Does Stage-1 Truly Matter?

So, where does this leave us in the grand scheme of VLM training? It seems that Stage-1 primarily impacts the entropy regime, a factor that could have potential downstream effects. But with the benefits being modest and localized, it challenges the assumption that OPD is inherently a superior warm-start for RL.

The AI Act text specifies that we must understand the intricacies of these stages to make informed decisions on model training methodologies. Could it be that the quest for an ultimate warm-start method is more about the journey than the destination?

Brussels moves slowly. But when it moves, it moves everyone. Perhaps it's time we apply that same patience and scrutiny to our understanding of these AI training stages. As the field evolves, the subtle differences may well determine the next leap in AI capabilities.