Rethinking Reinforcement Learning for Vision-Language Models

Reinforcement learning (RL) is making waves, especially in the space of vision-language models (VLMs), where it promises to boost performance on reasoning-heavy tasks. Yet, while RL-tuned VLMs show progress on benchmarks, they still trip over basic hurdles like weak visual grounding and hallucinations.

Unmasking Weaknesses

Introducing simple textual disturbances, think misleading captions and incorrect chains-of-thought (CoT), can shake these models to their cores. Such disturbances lead to sharp declines in their robustness and confidence. Interestingly, this vulnerability is even more pronounced when considering CoT consistency across open-source multimodal reasoning models. Closed models face similar issues but show better resilience, hinting that current open-source RL fine-tuning might be dropping the ball.

But what does this reveal? It underscores a critical flaw in how we currently train and evaluate these AI systems. Most assessments still prioritize accuracy over the robustness and faithfulness of reasoning. Slapping a model on a GPU rental isn't a convergence thesis. Accuracy-only evaluations miss the point entirely.

The Trade-off Dilemma

Digging deeper into RL fine-tuning reveals a trade-off between accuracy and faithfulness. Sure, fine-tuning might raise benchmark scores, but it can simultaneously undermine the reliability of reasoning and its adaptability to new contexts. Adversarial augmentation might stiffen the model's defense, but it doesn't stop faithfulness from drifting.

One solution? A faithfulness-aware reward system that realigns answers with reasoning. But beware: when combined with augmentation, there's a risk of models defaulting to shortcut strategies, leaving robustness a distant dream. If the AI can hold a wallet, who writes the risk model?

Beyond Numbers

These revelations demand a shift in training and evaluation protocols. We need to go beyond just accuracy. Correctness, robustness, and the faithfulness of reasoning must be part of the equation. The intersection is real. Ninety percent of the projects aren't. Are we ready to challenge the status quo and refine how we measure AI's reasoning capabilities?