PaperlayerView exact revision on arXiv

Source-linked AI summary

Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learning

Shuang Chen, Yue Guo, Zhaochen Su, Yafu Li, Yulun Wu, Jiacheng Chen, Jiayu Chen, Weijie Wang, Xiaoye Qu, Yu Cheng

arXiv:2506.04207v2cs.LGcs.AIcs.CLcs.CV

TL;DR

Directly applying reinforcement learning to MLLMs has not reliably activated complex multimodal reasoning. This paper introduces ReVisual-R1, using a text-centric cold start followed by multimodal RL with PAD and text-only RL refinement. It reports superior performance among open-source 7B models on challenging benchmarks.

  • Problem

    Direct application of text-centric RL to MLLMs has yielded diminishing returns, leaving open how to develop abstract linguistic skill while grounding it in visual perception.

    Source: §1 · p. 2

  • Method

    ReVisual-R1 uses a three-stage curriculum: text-centric cold start, multimodal RL with PAD, then text-only RL refinement.

    Source: §1 · p. 2, §4.1 · p. 5

  • Results

    ReVisual-R1 achieves superior performance among open-source 7B models across challenging multimodal, textual-reasoning, and generalization benchmarks.

    Source: §7 · p. 11

  • Takeaways & Limitations

    The reported curriculum combines text-only reasoning development with multimodal grounding, and the final text-RL phase further enhances multimodal reasoning without eroding visual grounding.

    Source: §1 · p. 2

    The study lacks a rigorous theoretical foundation for its text-centric strategies and staged RL, and its scalability evidence is limited to mid-sized models.

    Source: §6.3 · p. 11

Abstract

from arXiv · show

Inspired by the remarkable reasoning capabilities of Deepseek-R1 in complex textual tasks, many works attempt to incentivize similar capabilities in Multimodal Large Language Models (MLLMs) by directly applying reinforcement learning (RL). However, they still struggle to activate complex reasoning. In this paper, rather than examining multimodal RL in isolation, we delve into current training pipelines and identify three crucial phenomena: 1) Effective cold start initialization is critical for enhancing MLLM reasoning. Intriguingly, we find that initializing with carefully selected text data alone can lead to performance surpassing many recent multimodal reasoning models, even before multimodal RL. 2) Standard GRPO applied to multimodal RL suffers from gradient stagnation, which degrades training stability and performance. 3) Subsequent text-only RL training, following the multimodal RL phase, further enhances multimodal reasoning. This staged training approach effectively balances perceptual grounding and cognitive reasoning development. By incorporating the above insights and addressing multimodal RL issues, we introduce ReVisual-R1, achieving a new state-of-the-art among open-source 7B MLLMs on challenging benchmarks including MathVerse, MathVision, WeMath, LogicVista, DynaMath, and challenging AIME2024 and AIME2025.

1 INTRODUCTION

Directly applying text-centric RL to MLLMs has not reliably elicited complex multimodal reasoning, motivating a closer examination of the training pipeline. The paper proposes a three-stage curriculum and reports that ReVisual-R1 outperforms larger public models on challenging benchmarks.

1 INTRODUCTION

  • The paper frames multimodal reasoning as a challenge of developing abstract linguistic skill while grounding it in visual perception, which naive co-training may fail to cultivate fully.The authors identify this as motivation for dissecting the MLLM training pipeline.Source: §1 · p. 2
  • Carefully selected text data for cold-start initialization can let an MLLM surpass many recent multimodal reasoning models before multimodal RL.The initialization is intended to instill foundational reflective capabilities and extended Chain-of-Thought reasoning.Source: §1 · p. 2
  • Standard GRPO in multimodal RL can suffer gradient stagnation, degrading training stability and performance; PAD filters zero-advantage samples and reweights informative trajectories.PAD is designed to focus learning on more impactful data and improve training stability.Source: §1 · p. 2
  • Text-only RL after multimodal RL further enhances multimodal reasoning while sharpening linguistic expression and logical consistency without eroding visual grounding.The paper describes this final stage as a polishing step that consolidates model capabilities.Source: §1 · p. 2
  • ReVisual-R1 significantly outperforms much larger public models across challenging benchmarks, with the framework also demonstrated at 3B scale.The reported evaluation includes MathVerse, MathVision, MathVista, DynaMath, WeMath, LogicVista, AIME24/25, GPQA, and MATH-500.Source: §1 · pp. 2–3
  • The proposed curriculum sequences text-centric cold start, multimodal RL with PAD, and text-only RL refinement to develop reasoning and visual grounding.The authors present ReVisual-R1 as a 7B-parameter open-source MLLM built around this curriculum.Source: §1 · p. 2

2 PRELIMINARIES

The preliminaries set up multimodal reasoning as reward-maximizing policy learning over visual and textual inputs, then describe GRPO's grouped relative-advantage approach. The section also situates these foundations within the paper's cold-start and staged-RL framework.

2.1 MULTIMODAL REASONING FORMULATION

  • The task input combines visual content and a textual query, and the model generates multi-step self-reflective reasoning toward a solution while maximizing expected reward.The reward is primarily rule-based: 1 for a correct answer and 0 otherwise.Source: §2.1 · p. 3

2.2 GROUP RELATIVE POLICY OPTIMIZATION

  • GRPO groups samples and uses a group-specific advantage in its objective; that advantage compares a response with group performance, normalized by group reward variance.The formulation maintains both policy and reference models for each group.Source: §2.2 · p. 3

2 PRELIMINARIES

  • ReVisual-R1's framework follows data collection and curation with cold start and staged reinforcement learning.The section introduces the task setting and the base training algorithm used in the method.Source: §2 · p. 3, Figure 2 · p. 4

3 OPTIMIZED COLD START FOR MULTIMODAL REASONING

The section finds that text-only cold-start data improves both textual and multimodal reasoning, then introduces GRAMMAR, a curated dataset for cold start and verifiable-reward RL.

3.1 PRELIMINARY STUDY OF COLD START

  • Text-only cold-start models improve textual and multimodal reasoning, while Vision-R1 and R1-One-Vision show limited gains; Figure 3 compares absolute gains across four tasks.The preliminary study fine-tuned Qwen2.5-VL-7B-Instruct on 40,000 sampled instances and evaluated on MathVerse, MathVision, AIME24, and Math500.Source: §3.1 · p. 4, Figure 3 · p. 4
  • DeepMath prompts averaged 8,207.76 tokens versus Vision-R1’s 821.48, while Vision-R1’s pass rate was 96.00% versus DeepMath’s 75.0%.The authors cite these findings as indicating that current multimodal cold-start datasets may lack sufficient complexity, and therefore use text-only cold-start data.Source: §3.1 · p. 5

3.2 GRAMMAR: GENERALIZED MULTIMODAL REASONING DATASET

  • GRAMMAR contains 283K complex textual cold-start samples, plus 31K textual and 21K multimodal examples for verifiable-reward reinforcement learning.The cold-start samples feature explicit reasoning paths; the RL examples are annotated with ground truths.Source: §3.2 · p. 5
  • GRAMMAR curation filters for verifiable answers, assesses question difficulty, then clusters and balances samples across topics and difficulty levels.The pipeline uses Qwen2.5-VL-7B-Instruct for initial pruning, Qwen2.5-VL-32B-Instruct for difficulty classification, and NV-Embedding-V2 with HDBSCAN for clustering.Source: §3.2 · p. 5

4 STAGED REINFORCEMENT OPTIMIZATION (SRO)

SRO sequences multimodal RL and text-only RL after cold start, using PAD to address GRPO gradient stagnation during multimodal training. The text-only stage aims to restore fluency and refine reasoning.

4 STAGED REINFORCEMENT OPTIMIZATION (SRO)

  • SRO comprises multimodal RL followed by textual RL to cultivate robust reasoning and diverse MLLM competencies.It follows a cold-start stage.Source: §4 · p. 5

4.1 STAGE 1: MULTIMODAL RL

  • Multimodal RL grounds textual concepts in visual information and supports cross-modal reasoning, using GRPO with PAD to improve gradient quality.This phase primarily uses multimodal samples from GRAMMAR and is intended to address complex-task and sparse-reward challenges.Source: §4.1 · p. 5

4.1.1 PRIORITIZED ADVANTAGE DISTILLATION (PAD)

  • PAD addresses GRPO gradient stagnation: uniform rewards under sparse binary rewards can produce null advantages and zero policy gradients, while selective sampling improves stability and efficiency.It filters samples with near-zero advantages, selects informative sequences within an advantage range, and prioritizes sampling by absolute advantage; sampling temperature may be decayed during training.Source: §4.1.1 · pp. 5–6

4.2 STAGE 2: TEXTUAL RL

  • Textual RL follows multimodal RL to restore and enhance linguistic fluency and refine reasoning through instruction-following, conversational, and complex text-problem-solving data.The phase uses GRPO augmented with PAD.Source: §4.2 · p. 6

5 EXPERIMENTS

Experiments evaluate ReVisual-R1 across multimodal and textual reasoning benchmarks, compare its three-stage curriculum and PAD with alternatives, and report strong results at 7B and 3B scales.

5.1 EXPERIMENTS SETUP

  • The evaluation spans multimodal mathematics, general multimodal reasoning, and text reasoning, using pass@1 except average@32 for AIME24/25.Baselines include commercial models, Qwen2.5-VL-7B, and specialized open-source reasoning MLLMs.Source: §5.1 · pp. 6–7

5.2 IMPLEMENTATION DETAILS

  • ReVisual-R1 uses Qwen2.5-VL-Instruct backbones; MRL omits the GRPO KL constraint, while TRL freezes the vision tower and adds a small KL penalty.Experiments used eight NVIDIA A100-80G GPUs.Source: §5.2 · p. 7

5.3 TRAINING DATASETS

  • Training uses approximately 283k text entries for cold start, 26k GRAMMAR multimodal entries for MRL, and 30k text entries for TRL.The TRL data are intended to refine nuanced understanding and generation capabilities.Source: §5.3 · p. 7

5.4 MAIN RESULTS

  • 53.1% average score, +16.8 percentage points over Table 2 baselines; ReVisual-R1-7B ranked first among open-source models on nine of ten benchmarks.The reported benchmark set includes both textual and multimodal tasks.Source: §5.4 · p. 7, Table 2 · p. 8
  • ReVisual-R1's average score (53.1%) exceeded OpenAI-GPT-4o (41.6%), while doubao-1.5-vision-pro-32k averaged 55.8%; ReVisual-R1-3B beat VLAA-Thinker-3B by 16.0% on average.On MATH500, ReVisual-R1 scored 89.2%, versus doubao-1.5-vision-pro-32k at 85.2% and OpenAI-GPT-4o at 74.6%.Source: §5.4 · p. 7

5.5.1 ABLATION ON SRO

  • 49.6 Avg was highest for CS + MRL + TRL; CS + MRL scored 47.7, reversed ordering 45.5, and mixed RL 47.6.CS + MRL scored 71.9 on MathVista despite its lower overall average.Source: §5.5.1 · pp. 8–9, Table 3 · p. 8

5.5.2 ABLATION STUDY ON PAD

  • Full PAD outperformed GRPO-Baseline, GRPO-Filter-only, Random-Sampling, and DAPO; its sampling strategy yielded higher reward accuracy and faster convergence.The study attributes PAD's benefit to effective sample filtering and prioritized sub-sampling.Source: §5.5.2 · pp. 9–10, Table 4 · p. 9, Figure 5 · p. 10

5.5.3 GENERALIZATION OF REVISUAL-R1

  • ReVisual-R1 also performed consistently well on general multimodal benchmarks MMMU, MMMU-Pro, CMMMU, and MMStar, and on textual MMLU.The paper presents Figure 4 and Table 6 for these generalization results.Source: Figure 4 · p. 9, §5.5.3 · p. 10

6 RELATED WORK

The related-work discussion situates multimodal reasoning among MLLM and RL developments, while the limitations section identifies theoretical and scalability gaps and future data-curriculum questions.

6.1 MULTIMODAL LARGE LANGUAGE MODEL

  • Prior open-source MLLM reasoning work includes chain-of-thought adaptations and supervised fine-tuning with reasoning traces; this paper introduces longer reflective responses for long-CoT reasoning.The discussion says many existing reasoning MLLMs produce relatively short responses, which can curtail reflection, visual exploration, and deep multimodal reasoning.Source: §6.1 · pp. 10–11

6.2 REINFORCEMENT LEARNING IN REASONING

  • Multimodal RL remains nascent, with early efforts focused on subdomains such as mathematics or generative reward models, often using commercial-model data.The section places these efforts alongside broader LLM work on direct RL, cold-start datasets, GRPO, and its refinements.Source: §6.2 · p. 11

6.3 LIMITATIONS AND FUTURE WORK

  • The study lacks a rigorous theoretical account and tests scalability only on 3B and 7B models, leaving larger architectures and other foundation-model families unvalidated.Future work will examine how multimodal-versus-text data ratio and quality across training stages affect curricula for diverse skill clusters.Source: §6.3 · p. 11

7 CONCLUSION

ReVisual-R1 combines text-only cold start, multimodal RL stabilized by PAD, and text-RL refinement; the authors report strong benchmark performance among open-source 7B models. The conclusion also describes reproducibility provisions and ethical safeguards.

7 CONCLUSION

  • ReVisual-R1’s three-stage curriculum yields superior performance among open-source 7B models on challenging multimodal, textual reasoning, and generalization benchmarks.The stages are a high-difficulty text-only cold start, multimodal RL with PAD, and final TextRL refinement.Source: §7 · p. 11
  • The authors provide an anonymous code repository, public training and evaluation datasets, and experimental hyperparameters in the appendix.Source: REPRODUCIBILITY STATEMENT · p. 12
  • The work uses public academic datasets, filters GRAMMAR to exclude sensitive information, and involves no human subjects.The authors foresee no direct societal risks or harmful applications.Source: ETHICS STATEMENT · p. 12

B Training settings

The authors state that an LLM assisted with grammatical accuracy and sentence structure, while the intellectual contributions remained theirs.

B Training settings

  • LLM assistance was limited to grammar and sentence-structure refinement; the authors attribute ideas, analysis, and manuscript composition to themselves.Source: A LLM USAGE STATEMENT · p. 18

B TRAINING SETTINGS

Training comprises cold start, multimodal reinforcement learning, and text-only reinforcement learning, with phase hyperparameters detailed in Table 5.

B TRAINING SETTINGS

  • Training has three phases—cold start, multimodal RL, and text-only RL—and Table 5 lists their key hyperparameters.Source: B TRAINING SETTINGS · p. 18

C ALGORITHM IN PRIORITIZED ADVANTAGE DISTILLATION (PAD)

PAD filters out near-zero-advantage samples, then prioritizes informative sequences when forming a distilled minibatch. Its algorithm description specifies the filtering and sampling steps.

C ALGORITHM IN PRIORITIZED ADVANTAGE DISTILLATION (PAD)

D REASONING EXAMPLE

The displayed PAD procedure filters sequences, prioritizes sampling from the effective set, and forms a distilled mini-batch; the accompanying example shows iterative reasoning followed by a summary answer.

  • The algorithm takes an original batch and its advantage estimates as inputs.Source: D REASONING EXAMPLE · p. 20
  • PAD retains sequences whose absolute advantages fall within an informative range bounded by Tlow and Thigh, excluding near-zero-advantage samples.Source: D REASONING EXAMPLE · p. 20
  • For the effective set, PAD assigns sampling probabilities with a temperature-controlled Softmax and uses a uniform distribution if the normalization term is zero.Source: D REASONING EXAMPLE · p. 20
  • PAD samples k' indices from the effective set according to those probabilities and assembles the selected sequences into a distilled mini-batch.Source: D REASONING EXAMPLE · p. 20
  • The example depicts long responses that hypothesize, reflect, verify, and correct before reaching a final answer and providing a summary answer.Source: Figure 6 · p. 21
Loading 2506.04207v2…