PaperlayerView exact revision on arXiv

Source-linked AI summary

VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning

Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, Wenhu Chen

arXiv:2504.08837v3cs.LGcs.AI

TL;DR

Multimodal reasoning models do not reliably develop the slow, reflective behavior seen in text-only systems, motivating direct reinforcement learning without distillation. The paper combines GRPO with Selective Sample Replay and Forced Rethinking, producing VL-Rethinker, which improves multimodal benchmark performance while leaving gaps on broader tasks and in understanding modality differences.

  • Problem

    Vision-language models remain comparable to fast-thinking models in multimodal reasoning, while explicit self-reflection does not consistently emerge through standard reinforcement learning.

  • Method

    The paper combines direct reinforcement learning with GRPO, Selective Sample Replay for vanishing advantages, and Forced Rethinking during rollouts.

  • Results

    VL-Rethinker improves performance across challenging multimodal benchmarks, including an 8.46% relative improvement on MathVision over 2.49% for passively forcing the baseline to rethink.

  • Takeaways & Limitations

    The results support slow-thinking reinforcement learning as an effective approach for improving vision-language reasoning and self-reflection.

  • Takeaways & Limitations

    The models still lag human expert performance on general multimodal tasks such as EMMA and MEGA-Bench, possibly because high-quality multimodal training data is lacking.

Abstract

from arXiv · show

Recently, slow-thinking systems like GPT-o1 and DeepSeek-R1 have demonstrated great potential in solving challenging problems through explicit reflection. They significantly outperform the best fast-thinking models, such as GPT-4o, on various math and science benchmarks. However, their multimodal reasoning capabilities remain on par with fast-thinking models. For instance, GPT-o1's performance on benchmarks like MathVista, MathVerse, and MathVision is similar to fast-thinking models. In this paper, we aim to enhance the slow-thinking capabilities of vision-language models using reinforcement learning (without relying on distillation) to advance the state of the art. First, we adapt the GRPO algorithm with a novel technique called Selective Sample Replay (SSR) to address the vanishing advantages problem. While this approach yields strong performance, the resulting RL-trained models exhibit limited self-reflection or self-verification. To further encourage slow-thinking, we introduce Forced Rethinking, which appends a rethinking trigger token to the end of rollouts in RL training, explicitly enforcing a self-reflection reasoning step. By combining these two techniques, our model, VL-Rethinker, advances state-of-the-art scores on MathVista, MathVerse to achieve 80.4%, 63.5% respectively. VL-Rethinker also achieves open-source SoTA on multi-disciplinary benchmarks such as MathVision, MMMU-Pro, EMMA, and MEGA-Bench, narrowing the gap with OpenAI-o1. Our empirical results show the effectiveness of our approaches.

1 Introduction

Slow-thinking language models excel at challenging math and science problems, but their multimodal reasoning remains comparable to fast-thinking systems. VL-Rethinker addresses this gap with direct reinforcement learning, Selective Sample Replay, and Forced Rethinking to promote effective multimodal reflection.

  • 73.9% on MathVista and 57.0% on MathVerse leave GPT-o1 slightly below Qwen2.5-VL-72B at 74.8% and 57.2%, respectively.
  • Direct reinforcement learning trains multimodal reasoning models without relying on distillation from stronger teacher models.
  • Selective Sample Replay augments GRPO with high-value experiences from past iterations to counteract vanishing advantages and stabilize gradient signals.
  • Forced Rethinking appends a textual trigger to rollout responses, prompting self-reflection and self-verification before the final answer.
  • VL-Rethinker achieves state-of-the-art results across mathematical and general-purpose multimodal benchmarks, including open-source state of the art on EMMA and MMMU-Pro.
  • RL-induced slow-thinking develops less consistently on vision-language tasks than on math-focused tasks, and the authors identify understanding this discrepancy as future work.

2 Preliminaries

The paper formulates multimodal reasoning as generating a correct textual response from image-and-query inputs and trains an instruction-tuned VLM with reward-based policy optimization. GRPO compares grouped responses, normalizes their rewards, and uses clipped updates to control policy deviation.

  • 2.1 Problem Formulation: A multimodal reasoning task maps one or more images and a textual query to a response that correctly integrates visual and textual information.
  • 2.1 Problem Formulation: The VLM is shifted from shallow fast-thinking responses toward deeper reasoning through direct reinforcement learning that rewards accurate, thorough outputs.
  • 2.1 Problem Formulation: The policy πθ(y|x) is trained to maximize expected reward over a dataset of multimodal queries and answers, using a binary correctness reward.
  • 2.2 Group Relative Policy Optimization (GRPO): GRPO estimates each response’s advantage by comparing its reward with other responses generated for the same query.
  • 2.2 Group Relative Policy Optimization (GRPO): The GRPO objective uses a clipped surrogate loss whose clip function keeps the current-to-reference policy ratio within a predefined range.

3 Our Method

The method combines Selective Sample Replay to stabilize GRPO training with Forced Rethinking to cultivate deliberate self-reflection in vision-language models.

  • Selective Sample Replay: GRPO suffers from Vanishing Advantages when all responses in a query group receive identical rewards, producing zero advantage signals.These groups no longer provide effective policy gradients.
  • Selective Sample Replay: As training progresses, Qwen2.5-VL-72B examples with non-zero advantages decline from approximately 40% initially to below 20% after 16 × 16 gradient steps.The decline reflects increasingly uniform rewards within query groups.
  • Selective Sample Replay: Selective Sample Replay stores explored trajectories with non-zero advantages and augments current batches with prioritized rehearsal samples from past iterations.Samples are prioritized by the absolute magnitude of their advantages, with α controlling prioritization intensity.
  • Selective Sample Replay: SSR counteracts vanishing advantages by emphasizing valuable experiences, providing more consistent gradient signals and stabilizing training.It dynamically prioritizes experiences near the model’s decision boundaries rather than using a static offline curriculum.
  • Forced Rethinking: Standard RL with GRPO and SSR does not consistently produce deliberate patterns such as explicit self-correction in vision-language models.Forced Rethinking is introduced to proactively encourage more extensive internal deliberation before final answers.
  • Forced Rethinking: Forced Rethinking appends a textual trigger after an initial rollout response, while applying the intervention to only a fraction q < 1 of responses and retaining correct rethinking trajectories.The method also uses a prompt hint and an additional SFT loss based on successful forced-rethinking trajectories.

4 Experiments

The experiments evaluate method effectiveness, ablations, and learned rethinking behavior across challenging multimodal benchmarks. VL-Rethinker improves performance, while SSR stabilizes training and Forced Rethinking promotes adaptive self-reflection.

  • Experimental Setup: The evaluation spans math reasoning, multidisciplinary understanding, and long-tailed real-world multimodal tasks, reporting Pass@1 accuracy with greedy decoding.The benchmarks include MathVista, MathVerse, MathVision, MMMU, MMMU-Pro, EMMA, and MegaBench.
  • Main Results: VL-Rethinker-72B improves over Qwen2.5-VL-72B and achieves state-of-the-art math-benchmark results among the compared models, including OpenAI-o1.VL-Rethinker-7B also substantially outperforms competing RL-trained 7B models.
  • Data Ablation: Removing scientific queries causes a pronounced performance drop, whereas removing text-only queries produces no significant difference.The ablation supports the importance of scientific data for general reasoning ability.
  • SSR Ablation: GRPO-SSR consistently outperforms GRPO and GRPO-Filter, addressing vanishing advantages through filtering and selective replay.GRPO-SSR redistributes training emphasis toward examples with large absolute advantages, improving training stability and convergence.
  • Forced Rethinking: Forced rethinking improves baseline performance, while training with Forced Rethinking yields stronger gains than forcing rethinking only at test time.On MathVision, VL-Rethinker achieves an 8.46% relative improvement versus 2.49% for passively forced baseline rethinking.
  • Learned Rethinking Behavior: VL-Rethinker spontaneously rethinks at varying rates across benchmarks rather than applying rethinking to every instance.The variation suggests selective triggering based on perceived difficulty or initial confidence.

5 Related Work

Related work covers multimodal instruction tuning, reinforcement-learning approaches to reasoning, and multimodal RL pipelines. The paper positions its approach as a more direct RL-only alternative to supervision- and distillation-heavy methods.

  • Multimodal Instruction Tuning: Instruction-tuning research aligns vision-language models with natural-language instructions using increasingly diverse and larger datasets.Examples progress from InstructBLIP, LLaVA, and MiniGPT-4 to Llava-OV, Infinity-MM, MAmmoTH-VL, and VisualWebInstruct.
  • RL for Reasoning: Recent work applies reinforcement learning directly to language models or modifies GRPO to elicit deeper reasoning behaviors.Examples include SimpleRL-Zoo, Open-Reasoner-Zero, DeepScaler, Light-R1, DAPO, and Dr GRPO.
  • Multimodal RL Pipelines: Multimodal reasoning methods commonly combine supervised fine-tuning on reasoning traces with subsequent reinforcement learning.These pipelines may require visual captioning, teacher-model distillation, and tightly coupled SFT-plus-RL orchestration.
  • Paper Positioning: This paper investigates a direct RL-only approach intended to incentivize slow-thinking behavior without large-scale supervision or teacher distillation.The contrast is with complex, resource-intensive multimodal reasoning pipelines.

6 Conclusion

The paper concludes that its approaches effectively incentivize multimodal reasoning, while acknowledging remaining performance gaps on broader multimodal tasks.

  • Limitations: The models still lag behind human expert performance on general multimodal tasks such as EMMA and MEGA-Bench.The authors conjecture that insufficient high-quality multimodal training data contributes to this gap.

A Training Dataset

The training dataset combines public multimodal resources with web-collected queries and is organized to support efficient training across model scales.

  • Dataset Construction: The seed dataset aggregates publicly available multimodal datasets with novel web-collected queries and has broad topical diversity.Figure 8 depicts eight major topic categories in the training data.
  • Scale-Specific Curation: Training subsets are tailored to each model scale using queries where the initial checkpoint has non-zero PassRate@8.This criterion targets queries within each model’s potential capabilities.

B Prompts

The paper uses separate prompts for initial RL training and Forced Rethinking. The latter explicitly encourages self-questioning, self-verification, and self-correction through textual triggers.

  • Default Instruction Prompt: The default instruction prompt asks the model to reason step by step and place its final answer within \boxed{}.
  • Default Instruction Prompt: During the first-stage RL training with SSR, the model uses the default instruction prompt.
  • Rethinking Instruction Prompt: The Rethinking Instruction Prompt asks the model to perform regular self-questioning, self-verification, and self-correction while reasoning.
  • Rethinking Instruction Prompt: During Forced Rethinking training, the model uses the rethinking prompt and three types of textual triggers to encourage self-reflection.
  • Rethinking Instruction Prompt: The three triggers prompt the model to check correctness, detect possible mistakes, and double-check its reasoning.
Loading 2504.08837v3…