Reviving Plasticity: Enhancing RL with Rejuvenation

Supervised Fine-Tuning (SFT) has become a familiar first step in refining Large Language Models (LLMs). However, it's often followed by Reinforcement Learning (RL) to enhance functionality. The current problem? Excessive SFT seems to impede RL by making models less adaptable.

The Plasticity Problem

Research indicates that overly fine-tuned models lose what experts call 'plasticity'. That's the ability for a model to be reshaped during RL. Why is this significant? The data shows that when SFT is overdone, models produce overly confident token distributions and rigid parameter landscapes. These factors complicate their further optimization in the RL phase.

Introducing Rejuvenation

Enter the proposed solution: Rejuvenation. This approach isn't just another buzzword. It combines base-anchored model fusion with a targeted neuron reset technique. The goal? To reinstate plasticity while maintaining valuable priors acquired during SFT. What the English-language press missed: this method effectively reduces the drift caused by excessive SFT.

Proven Results

The benchmark results speak for themselves. Experiments on both mathematical reasoning and agentic tasks demonstrate that Rejuvenation consistently improves RL performance in models burdened by excessive SFT. Not only does it enhance performance, but it also boosts generalization to out-of-distribution tasks.

: Why hasn't Rejuvenation been more widely adopted? Western coverage has largely overlooked this innovation. As AI continues to evolve, methods like Rejuvenation could become vital in maintaining the delicate balance between SFT and RL. In the competitive world of AI development, optimization is everything.