Rethinking Evolution Strategies for Language Model Fine-Tuning

Evolution Strategies (ES) has recently stepped into the spotlight as an appealing contender for fine-tuning large language models (LLMs), often rivaling traditional reinforcement learning (RL) methods. The draw? Simplicity, scalability, and the notable advantage of inference-only training. Yet, a lingering concern is the issue of task forgetting when introducing new tasks.

Misconceptions About Forgetting

At first glance, it seemed ES had a problem. The perception was that in the pursuit of new tasks, the system would forget what it had already learned. However, recent insights suggest this is more of a performance drift than an irreversible forgetting. Prior tasks aren't lost forever. they often rebound during the ES training process. What's even more intriguing is that this isn't just an ES issue. RL methods, too, aren't immune to the same fate.

The Dynamics Behind Performance Drift

So, why does this drift occur? It boils down to the training dynamics inherent in ES, especially the random walks in poorly constrained directions of the weight space. This isn't merely a technical curiosity. It raises the question: are we overlooking the potential of ES due to a misunderstanding of its behavior?

Enter Anchored Weight Decay (AWD), a big deal in this narrative. By introducing a parameter-space regularization technique, AWD anchors optimization closer to the original model parameters. This approach effectively curbs performance drift, ensuring that prior-task performance remains stable while the model continues to excel in new tasks. In essence, ES has been rehabilitated from the shadows, thanks to AWD.

The Case for ES in Continual Learning

The AI-AI Venn diagram is getting thicker, as ES now stands as a formidable approach for continual learning within LLMs. The stabilization offered by AWD means that the benefits of large ES population sizes can be reaped at a fraction of the computational cost. In an industry where efficiency and efficacy are important, this revelation is a significant stride forward.

Critics who previously dismissed ES might need to reassess their stance. If the issue of prior-task forgetting is largely avoidable, then why not embrace the simplicity and scalability ES offers? As the industry pushes for continuous learning, ES, armed with methodologies like AWD, might just be the future we've been overlooking.