IRDS: Elevating Reinforcement Learning with Verifiable Rewards

Reinforcement learning with verifiable rewards (RLVR) is on the rise, especially in enhancing large language model (LLM) reasoning. However, the journey hasn't been smooth. Data inefficiency stands out as a persistent challenge. Traditional methods have tried to address this, but they often fall short. They miss vital components like subset-level coverage, verifier signal use, or interpretability.

Introducing IRDS

This is where IRDS (Interpretable RLVR Data Selection) makes a mark. By focusing on a sparse autoencoder (SAE) cluster basis, IRDS carefully selects RLVR training instances. The selection process itself is auditable, focusing on recognizable problem motifs. The magic lies in choosing instances where the model fails yet has room to learn. This is achieved through a verifier-coupled coverage objective, optimized by greedy log-determinant maximization.

Performance Metrics

The results? They're impressive. IRDS was tested on three instruction-tuned models across six math reasoning benchmarks. It achieved the highest overall accuracy. Specifically, it surpassed the strongest baseline by +3.9/+4.0 percentage points on the two Qwen models and by +0.5 percentage points on Llama-3.1-8B. Cost-wise, IRDS runs an order of magnitude cheaper than its trajectory-based counterparts.

Why It Matters

So, why does this matter? The chart tells the story. In an industry where efficiency often comes at the expense of interpretability, IRDS provides a balanced solution. It offers a tangible improvement in performance without breaking the bank. Visualize this: higher accuracy with reduced costs.

But let's pause and consider: is this the end-all solution to RLVR's inefficiencies? Likely not. Yet, IRDS sets a new benchmark, challenging existing norms and pushing the boundaries of what's possible. Is it the model's interpretability or the cost-effectiveness that stands out? The trend is clearer when you see it.

In the relentless pursuit of better machine learning models, IRDS offers a glimpse into a future where data selection isn't just about volume but about verifiable quality. Numbers in context: IRDS doesn't just improve metrics. it redefines how we approach data selection in reinforcement learning.