Rethinking Replay Buffers in LLM Post-Training: A Case for Efficiency

Experience replay has long been a staple in the reinforcement learning toolkit. But it's a different story for large language models (LLMs). The prevailing wisdom insists on fresh, on-policy data to reach peak performance. But does the data need to be fresh to be effective?

Challenging Conventional Wisdom

Recent research challenges the notion that on-policy data is essential for LLM post-training. The study presents a compelling argument for using replay buffers. The idea is to balance the trade-offs between the staleness of data, the diversity of samples, and the high computational costs of generating new data. Here's what the benchmarks actually show: strict on-policy sampling falls short when generation costs are high.

Efficiency Over Freshness

What does this mean for LLMs? It means there's a potential to drastically reduce inference compute without sacrificing model performance. In fact, a well-designed replay buffer can sometimes improve results while maintaining policy entropy. The numbers tell a different story from the traditional beliefs. So why hasn't this method been adopted more widely?

The Case for Replay Buffers

Replay buffers could redefine how we approach LLM post-training. They offer a method to cut down on computational resources while preserving, and occasionally enhancing, model capabilities. The reality is, especially with the high costs of data generation, efficiency should take precedence over the freshness of data. Strip away the marketing and you get a clearer picture: the architecture matters more than the parameter count. Isn't it time we rethink how we train our models?