PaperlayerView exact revision on arXiv

Source-linked AI summary

VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning

Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, Wenhu Chen

arXiv:2504.08837v3cs.LGcs.AI

TL;DR

Multimodal reasoning models do not reliably develop the slow, reflective behavior seen in text-only systems, motivating direct reinforcement learning without distillation. The paper combines GRPO with Selective Sample Replay and Forced Rethinking, producing VL-Rethinker, which improves multimodal benchmark performance while leaving gaps on broader tasks and in understanding modality differences.

  • Problem

    Vision-language models remain comparable to fast-thinking models in multimodal reasoning, while explicit self-reflection does not consistently emerge through standard reinforcement learning.

    Source: §1 · p. 2, How can we effectively incentivize… · p. 2, §3.3 · p. 6

  • Method

    The paper combines direct reinforcement learning with GRPO, Selective Sample Replay for vanishing advantages, and Forced Rethinking during rollouts.

    Source: How can we effectively incentivize… · p. 2, §2.2 · p. 3, §3.1 · p. 4

  • Results

    VL-Rethinker improves performance across challenging multimodal benchmarks, including an 8.46% relative improvement on MathVision over 2.49% for passively forcing the baseline to rethink.

    Source: §4.1 · p. 8, §4.2 · p. 9

  • Takeaways & Limitations

    The results support slow-thinking reinforcement learning as an effective approach for improving vision-language reasoning and self-reflection.

    Source: How can we effectively incentivize… · p. 2

    The models still lag human expert performance on general multimodal tasks such as EMMA and MEGA-Bench, possibly because high-quality multimodal training data is lacking.

    Source: §6 · p. 11

Abstract

from arXiv · show

Recently, slow-thinking systems like GPT-o1 and DeepSeek-R1 have demonstrated great potential in solving challenging problems through explicit reflection. They significantly outperform the best fast-thinking models, such as GPT-4o, on various math and science benchmarks. However, their multimodal reasoning capabilities remain on par with fast-thinking models. For instance, GPT-o1's performance on benchmarks like MathVista, MathVerse, and MathVision is similar to fast-thinking models. In this paper, we aim to enhance the slow-thinking capabilities of vision-language models using reinforcement learning (without relying on distillation) to advance the state of the art. First, we adapt the GRPO algorithm with a novel technique called Selective Sample Replay (SSR) to address the vanishing advantages problem. While this approach yields strong performance, the resulting RL-trained models exhibit limited self-reflection or self-verification. To further encourage slow-thinking, we introduce Forced Rethinking, which appends a rethinking trigger token to the end of rollouts in RL training, explicitly enforcing a self-reflection reasoning step. By combining these two techniques, our model, VL-Rethinker, advances state-of-the-art scores on MathVista, MathVerse to achieve 80.4%, 63.5% respectively. VL-Rethinker also achieves open-source SoTA on multi-disciplinary benchmarks such as MathVision, MMMU-Pro, EMMA, and MEGA-Bench, narrowing the gap with OpenAI-o1. Our empirical results show the effectiveness of our approaches.

1 Introduction

Slow-thinking language models excel at challenging math and science problems, but their multimodal reasoning remains comparable to fast-thinking systems. VL-Rethinker addresses this gap with direct reinforcement learning, Selective Sample Replay, and Forced Rethinking to promote effective multimodal reflection.

2 Preliminaries

The paper formulates multimodal reasoning as generating a correct textual response from image-and-query inputs and trains an instruction-tuned VLM with reward-based policy optimization. GRPO compares grouped responses, normalizes their rewards, and uses clipped updates to control policy deviation.

2.1 Problem Formulation

  • A multimodal reasoning task maps one or more images and a textual query to a response that correctly integrates visual and textual information.Source: §2.1 · p. 3
  • The VLM is shifted from shallow fast-thinking responses toward deeper reasoning through direct reinforcement learning that rewards accurate, thorough outputs.Source: §2.1 · p. 3
  • The policy πθ(y|x) is trained to maximize expected reward over a dataset of multimodal queries and answers, using a binary correctness reward.Source: §2.1 · p. 3

2.2 Group Relative Policy Optimization (GRPO)

  • GRPO estimates each response’s advantage by comparing its reward with other responses generated for the same query.Source: §2.2 · p. 3
  • The GRPO objective uses a clipped surrogate loss whose clip function keeps the current-to-reference policy ratio within a predefined range.Source: §2.2 · p. 3

3 Our Method

The method combines Selective Sample Replay to stabilize GRPO training with Forced Rethinking to cultivate deliberate self-reflection in vision-language models.

Selective Sample Replay

  • GRPO suffers from Vanishing Advantages when all responses in a query group receive identical rewards, producing zero advantage signals.These groups no longer provide effective policy gradients.Source: §3.1 · p. 4
  • As training progresses, Qwen2.5-VL-72B examples with non-zero advantages decline from approximately 40% initially to below 20% after 16 × 16 gradient steps.The decline reflects increasingly uniform rewards within query groups.Source: §3.1 · p. 4
  • Selective Sample Replay stores explored trajectories with non-zero advantages and augments current batches with prioritized rehearsal samples from past iterations.Samples are prioritized by the absolute magnitude of their advantages, with α controlling prioritization intensity.Source: §3.2 · pp. 4–5
  • SSR counteracts vanishing advantages by emphasizing valuable experiences, providing more consistent gradient signals and stabilizing training.It dynamically prioritizes experiences near the model’s decision boundaries rather than using a static offline curriculum.Source: §3.2 · p. 5

Forced Rethinking

  • Standard RL with GRPO and SSR does not consistently produce deliberate patterns such as explicit self-correction in vision-language models.Forced Rethinking is introduced to proactively encourage more extensive internal deliberation before final answers.Source: §3.3 · p. 6
  • Forced Rethinking appends a textual trigger after an initial rollout response, while applying the intervention to only a fraction q < 1 of responses and retaining correct rethinking trajectories.The method also uses a prompt hint and an additional SFT loss based on successful forced-rethinking trajectories.Source: §3.3 · p. 6

4 Experiments

The experiments evaluate method effectiveness, ablations, and learned rethinking behavior across challenging multimodal benchmarks. VL-Rethinker improves performance, while SSR stabilizes training and Forced Rethinking promotes adaptive self-reflection.

Experimental Setup

  • The evaluation spans math reasoning, multidisciplinary understanding, and long-tailed real-world multimodal tasks, reporting Pass@1 accuracy with greedy decoding.The benchmarks include MathVista, MathVerse, MathVision, MMMU, MMMU-Pro, EMMA, and MegaBench.Source: §4 · p. 7

Main Results

  • VL-Rethinker-72B improves over Qwen2.5-VL-72B and achieves state-of-the-art math-benchmark results among the compared models, including OpenAI-o1.VL-Rethinker-7B also substantially outperforms competing RL-trained 7B models.Source: Table 1 · p. 7, §4.1 · p. 8, Table 2 · p. 8

Data Ablation

  • Removing scientific queries causes a pronounced performance drop, whereas removing text-only queries produces no significant difference.The ablation supports the importance of scientific data for general reasoning ability.Source: §4.2 · p. 8, Table 3 · p. 8

SSR Ablation

  • GRPO-SSR consistently outperforms GRPO and GRPO-Filter, addressing vanishing advantages through filtering and selective replay.GRPO-SSR redistributes training emphasis toward examples with large absolute advantages, improving training stability and convergence.Source: §4.2 · p. 9

Forced Rethinking

  • Forced rethinking improves baseline performance, while training with Forced Rethinking yields stronger gains than forcing rethinking only at test time.On MathVision, VL-Rethinker achieves an 8.46% relative improvement versus 2.49% for passively forced baseline rethinking.Source: §4.2 · p. 9

Learned Rethinking Behavior

  • VL-Rethinker spontaneously rethinks at varying rates across benchmarks rather than applying rethinking to every instance.The variation suggests selective triggering based on perceived difficulty or initial confidence.Source: §4.2 · p. 10

5 Related Work

Related work covers multimodal instruction tuning, reinforcement-learning approaches to reasoning, and multimodal RL pipelines. The paper positions its approach as a more direct RL-only alternative to supervision- and distillation-heavy methods.

Multimodal Instruction Tuning

  • Instruction-tuning research aligns vision-language models with natural-language instructions using increasingly diverse and larger datasets.Examples progress from InstructBLIP, LLaVA, and MiniGPT-4 to Llava-OV, Infinity-MM, MAmmoTH-VL, and VisualWebInstruct.Source: §5.1 · p. 10

RL for Reasoning

  • Recent work applies reinforcement learning directly to language models or modifies GRPO to elicit deeper reasoning behaviors.Examples include SimpleRL-Zoo, Open-Reasoner-Zero, DeepScaler, Light-R1, DAPO, and Dr GRPO.Source: §5.2 · pp. 10–11

Multimodal RL Pipelines

  • Multimodal reasoning methods commonly combine supervised fine-tuning on reasoning traces with subsequent reinforcement learning.These pipelines may require visual captioning, teacher-model distillation, and tightly coupled SFT-plus-RL orchestration.Source: §5.3 · p. 11

Paper Positioning

  • This paper investigates a direct RL-only approach intended to incentivize slow-thinking behavior without large-scale supervision or teacher distillation.The contrast is with complex, resource-intensive multimodal reasoning pipelines.Source: §5.3 · p. 11

6 Conclusion

The paper concludes that its approaches effectively incentivize multimodal reasoning, while acknowledging remaining performance gaps on broader multimodal tasks.

Limitations

  • The models still lag behind human expert performance on general multimodal tasks such as EMMA and MEGA-Bench.The authors conjecture that insufficient high-quality multimodal training data contributes to this gap.Source: §6 · p. 11

A Training Dataset

The training dataset combines public multimodal resources with web-collected queries and is organized to support efficient training across model scales.

Dataset Construction

  • The seed dataset aggregates publicly available multimodal datasets with novel web-collected queries and has broad topical diversity.Figure 8 depicts eight major topic categories in the training data.Source: A Training Dataset · p. 16, Figure 8 · p. 16

Scale-Specific Curation

  • Training subsets are tailored to each model scale using queries where the initial checkpoint has non-zero PassRate@8.This criterion targets queries within each model’s potential capabilities.Source: A Training Dataset · p. 16

B Prompts

The paper uses separate prompts for initial RL training and Forced Rethinking. The latter explicitly encourages self-questioning, self-verification, and self-correction through textual triggers.

Default Instruction Prompt

  • The default instruction prompt asks the model to reason step by step and place its final answer within \boxed{}.Source: B Prompts · p. 16
  • During the first-stage RL training with SSR, the model uses the default instruction prompt.Source: B Prompts · p. 16

Rethinking Instruction Prompt

  • The Rethinking Instruction Prompt asks the model to perform regular self-questioning, self-verification, and self-correction while reasoning.Source: B Prompts · p. 17
  • During Forced Rethinking training, the model uses the rethinking prompt and three types of textual triggers to encourage self-reflection.Source: B Prompts · p. 17
  • The three triggers prompt the model to check correctness, detect possible mistakes, and double-check its reasoning.Source: B Prompts · p. 17

Related papers

Loading 2504.08837v3…