PaperlayerView exact revision on arXiv

Source-linked AI summary

Diving into Self-Evolving Training for Multimodal Reasoning

Wei Liu, Junlong Li, Xiwen Zhang, Fan Zhou, Yu Cheng, Junxian He

arXiv:2412.17451v3cs.CLcs.AIcs.CVcs.LG

TL;DR

Multimodal self-evolving training remains underexplored despite limited annotated thought processes and the need to understand its design factors. The paper analyzes the approach through reinforcement learning and reports M-STAR gains across benchmarks and model sizes.

  • Problem

    Multimodal self-evolving training has limited prior study and no unified framework, while human-annotated thought processes for multimodal scenarios remain scarce.

    Source: §1 · p. 1

  • Method

    The paper analyzes self-evolving training through reinforcement learning, studies training method, reward model, prompt variation, and dynamics, and combines its design findings in M-STAR.

    Source: §1 · p. 1

  • Results

    M-STAR reports performance gains across five multimodal reasoning benchmarks for models ranging from 2B to 8B parameters.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper concludes that balancing exploration and exploitation can further improve self-evolving training for multimodal reasoning.

    Source: §5 · pp. 8–9

    Reward-Pass@2 is an imperfect proxy for top-two chain-of-thought quality because no reliable method to score thought-process quality is available.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

Self-evolving trainin--where models iteratively learn from their own outputs--has emerged as a key approach for complex reasoning tasks, addressing the scarcity of high-quality chain-of-thought data. However, its effectiveness in multimodal reasoning, a domain more intricate than text-only reasoning, remains underexplored, and the understanding of critical factors in this training paradigm remains limited. Furthermore, a central challenge for this training method is performance saturation, which impedes further improvements and scalability. Inspired by reinforcement learning (RL), in this paper, we reframe self-evolving training for multimodal reasoning through the lens of RL, identifying three pivotal factors: Training Method, Reward Model, and Prompt Variation. Through systematic analysis, we establish relatively optimal design principles that significantly enhance multimodal reasoning capabilities. Moreover, delving deeper into training dynamics, we uncover the roots of saturation and propose a new automatic balancing mechanism to mitigate this limitation. Building on these insights, we propose M-STAR (Multimodal Self-evolving Training for Reasoning), a framework that achieves consistent performance gains across models of varying sizes and diverse benchmarks. All resources are made publicly available at https://mstar-lmm.github.io.

1. Introduction

The paper reframes multimodal self-evolving training through reinforcement learning, studies its design choices and dynamics, and proposes M-STAR. It reports gains across benchmarks and model sizes.

1. Introduction

  • Multimodal reasoning matters in real-world applications, yet annotated thought processes are scarce and multimodal self-evolving research lacks a unified framework.The paper describes multimodal reasoning as requiring understanding beyond text, including analysis of figures, diagrams, and charts.Source: §1 · p. 1
  • Controlled studies identify continuous self-evolving training, a multimodal process reward model, and conditional use of unlabeled queries as design findings.Continuous training outperforms other iterative baselines; more unlabeled queries help with perfect reward signals but can hurt when a reward model generalizes poorly to unseen data.Source: §1 · p. 1
  • Performance saturation is attributed to diminishing exploration potential; the paper introduces an exploration–exploitation metric and adaptive sampling-temperature balancing.The mechanism dynamically adjusts sampling temperature to sustain the exploration–exploitation trade-off.Source: §1 · p. 1
  • Figure 1 separates the static design factors—training method, reward model, and prompt variation—from monitored dynamics that provide training control signals.Source: Figure 1 · p. 2
  • M-STAR combines optimized design choices with dynamic adjustments and reports gains across five multimodal benchmarks for models from 2B to 8B.The benchmarks are MathVista, M3CoT, MMStar, MMBench, and AI2D; the evaluated models include InternVL2 (2B), Phi-3.5-Vision (4B), and MiniCPM-V-2.5 (8B).Source: §1 · pp. 1–2

2. Overview of Self-Evolving Training for Multimodal Reasoning

The overview frames self-evolving training as reinforcement learning and describes an iterative generate-and-improve process. It identifies training method, reward design, and prompt variation as key design spaces.

2. Overview of Self-Evolving Training for Multimodal Reasoning

  • Self-evolving training is framed as reinforcement learning that trains a policy to maximize expected reward.Source: §2 · p. 2
  • The iterative process separates response generation from improvement: the current policy samples responses, then rewards select responses for supervised fine-tuning.This separation trades off offline and online training.Source: §2 · p. 2
  • Ground-truth answers can supply rewards, while a reward model must score responses independently when answers are unavailable.Source: §2 · p. 2
  • The overview identifies three design spaces: the iterative training method, reward function, and use of unlabeled prompts without ground-truth answers.Training methods differ, for example, in whether they initialize from the latest or original checkpoint at each iteration.Source: §2 · p. 2

3. Diving into Self-Evolving Design Components

This section evaluates training methods, reward models, and prompt variation in multimodal self-evolving training. Continuous optimization and PRM-based selection improve results, while training with unlabeled prompts remains challenging.

3.1. General Setup

  • In the setup, MiniCPM-V-2.5 is the main analysis model; 180K MathV360K examples form the labeled training set, with MathVista testmini used for OOD evaluation.The authors also validate final design choices on Phi-3.5-Vision and InternVL-2.Source: §3.1 · p. 3

3.2. Training Methods

  • Continuous Self-Evolving inherits the model checkpoint, optimizer, and learning-rate scheduler between iterations, making optimization continuous and closer to online learning.The study varies the iteration interval, or number of training queries passed per iteration.Source: §3.2 · p. 3
  • 43.1% on the in-domain MathV360K test set and 57.2% on MathVista: continuous self-evolving achieves the best performance on both tests.The authors report that proper iteration intervals matter: large intervals delay updates, while switching too frequently makes learning unstable.Source: §3.2 · p. 3, Table 1 · p. 4

3.3. Reward Models

  • The PRM assigns scores to intermediate reasoning steps, aggregates them with a minimum operation, and is applied after responses are filtered for answer correctness.The PRM is trained using Monte Carlo rollouts from partial reasoning prefixes.Source: §3.3 · p. 4
  • In Table 2, Continuous Self-Evolving with PRM using Top-2 achieves the best performance in both the ID and OOD tests, scoring 45.3% and 59.2%, respectively.Compared with training without PRM, most instances of self-evolving training with PRM show improved performance, especially in the OOD test.Source: §3.3 · p. 4
  • The PRM performs poorly as a verifier but reranks correct responses toward fewer reasoning steps and greater relevance to the question.Threshold filtering can retain or discard all responses for a query, reducing diversity; the authors therefore favor PRM as a reranker.Source: §3.3 · p. 5

3.4. Prompt Variation

  • With oracle reward signals but no PRM, unlabeled prompts outperform labeled-only training on OOD evaluation but underperform on ID evaluation; adding PRM leaves models below the best labeled-only model on both.The authors attribute the PRM result to its inability to verify responses without ground-truth answers and concerns about generalization.Source: Table 3 · p. 6, §3.4 · p. 6
  • Introducing unlabeled prompts from the beginning mitigates their negative performance impact compared with introducing them midway, but insufficient surrogate supervision can harm training and shift the policy distribution.Introducing them later also reduces their overall participation in training.Source: §3.4 · p. 6

4. Dynamics of Self-Evolution & Final Recipe

The section examines how exploration and reward-based exploitation change during self-evolving training, then combines selected design choices with adaptive temperature tuning in M-STAR. M-STAR achieves the highest MathVista results for all three backbones and generally outperforms base and warmup models across the broader benchmark set, with an exception for InternVL2-2B.

4.1. Monitoring the Training Dynamics

  • As training progresses, Pass@K declines while greedy accuracy improves; larger temperatures resist the Pass@K decline more effectively in mid-to-late training.The authors interpret the pattern as loss of exploration that may contribute to performance saturation.Source: §4.1 · p. 7, Figure 4 · p. 7
  • Reward-Pass@2 initially rises but quickly plateaus, suggesting diminishing exploitation as training progresses.The authors identify reduced exploration and reward-model constraints as possible contributors.Source: §4.1 · p. 7

4.2. M-STAR– Final Recipe with Optimal Design Choices & Adaptive Explorations

  • Improving the reward model itself is left for future work.Source: §4.2 · p. 7
  • Reward-Pass@2 is an imperfect proxy because responses are filtered for correct answers before reward-model reranking, and no reliable method scores reasoning quality.The authors nevertheless describe it as a reasonable approximation that works empirically.Source: §4.2 · p. 7
  • M-STAR achieves the highest MathVista results for all three backbones and outperforms base and warmup models on nearly all broader benchmarks, with InternVL2-2B underperforming on two.Table 4 compares training strategies across model sizes; Table 5 covers additional benchmarks.Source: Table 4 · p. 7, Table 5 · p. 8, §4.2 · p. 8
  • M-STAR automatically adjusts temperature using validation Reward-Pass@2, mitigating Pass@K regression and generally raising Reward-Pass@2 relative to fixed-temperature training.The temperature is selected every two iterations from 0.3 to 1.6 in 0.1 increments.Source: §4.2 · pp. 7–8, Figure 5 · p. 8

5. Conclusion

The conclusion presents controlled study of three design factors and analysis of exploration–exploitation dynamics as the basis for improved multimodal self-evolving training. It identifies balancing training dynamics as a way to further improve performance and offers the findings as guidance for future research.

5. Conclusion

  • The paper identifies training method, reward model, and prompt variation as three key factors and uses controlled experiments to derive design choices.Source: §5 · p. 8
  • The conclusion connects analysis of the exploration–exploitation trade-off with improved performance through balancing training dynamics.Source: §5 · pp. 8–9
  • The authors present their findings as insights and guidance for future multimodal self-evolving training research.Source: §5 · p. 9

A. Details of Selected LMMs

The section profiles three selected LMMs, describing their architectures and multimodal capabilities.

A. Details of Selected LMMs

  • MiniCPM-V-2.5 combines LLaMA-3-8B with a SigLIP vision encoder and is described as surpassing earlier openly released LMMs.The passage identifies LLaVA and Qwen-VL as examples of those earlier models.Source: A. Details of Selected LMMs · p. 12
  • Phi-3.5-Vision combines a CLIP ViT-L/14 image encoder with a Phi-3.5-mini decoder and handles interleaved image-text inputs through dynamic image cropping.It was pretrained on 0.5T tokens and post-trained with SFT and DPO.Source: A. Details of Selected LMMs · p. 12
  • InternVL 2.0 is a 1B–108B model series; the 2B version used here combines InternViT, an MLP projector, and InternLM-2-Chat.The passage describes support for diverse inputs, including text, images, video, and medical data.Source: A. Details of Selected LMMs · p. 12

B. Warm-Up Phase to Unlock the Chain-of-Thought (CoT) Capability of LMMs

The warm-up phase addresses open-source LMMs’ difficulty generating detailed CoT, which the iterative training process needs. The authors elicit and fine-tune on model-generated reasoning with correct final answers.

B. Warm-Up Phase to Unlock the Chain-of-Thought (CoT) Capability of LMMs

  • Open-source LMMs often answer directly rather than produce detailed CoT, while self-evolving training needs varied intermediate steps from on-policy data.The authors associate the reasoning difficulty with scarce high-quality rationales in multimodal SFT datasets.Source: B. Warm-Up Phase to Unlock the… · p. 12
  • The warm-up prompt asks models to generate intermediate reasoning steps for a question, image, and answer rather than answer directly.The instruction requests a comprehensive analytical breakdown and places the answer at the end.Source: B. Warm-Up Phase to Unlock the… · p. 12
  • For each question-image-answer triplet, models generate 16 samples at temperature 1.0; incorrect-final-answer outputs are filtered before 100K examples are used for warm-up fine-tuning.The resulting CoT dataset is treated as a standard RFT process, and iterative training starts from that checkpoint.Source: B. Warm-Up Phase to Unlock the… · p. 12

C. Hyper Parameters

The section specifies the optimization and image-processing settings used in the training setup.

C. Hyper Parameters

  • Training uses a 1e-6 learning rate, batch size 128, constant scheduler with 0.1 warmup ratio, and SigLIP SoViT-400m/14 image encoding.Images are sliced into at most 9 segments, each compressed into 96 queries through a perceiver resampler with one cross-attention layer.Source: C. Hyper Parameters · p. 12

D. Training Process Reward Model (PRM)

The PRM is trained using sampled model responses and stepwise labels, then assessed as a re-ranker using response-relativity ratings.

D. Training Process Reward Model (PRM)

E. Measuring Response Relativity

F. Extra Comparison between RM-selected Solutions and Others

The process reward model ranks top-two solutions that typically use fewer reasoning steps, suggesting it can favor more straightforward solutions with fewer irrelevant steps.

G. More Results for M-STAR

Across selected temperatures, M-STAR’s exploration ability does not decline continuously during training, while Reward-Pass@2 reaches its peak more quickly than without dynamic monitoring.

H. Full Results of MathVista

M-STAR improves MathVista performance across subtasks, particularly geometry problems and math word problems; the accompanying related-work review situates self-evolving approaches among iterative and multimodal methods.

H. Full Results of MathVista

  • Figure 6 compares reasoning-step counts for the reward model’s top-two solutions with the remaining solutions.Source: Figure 6 · p. 13
  • M-STAR improves on the pre-evolved models across MathVista subtasks, particularly geometry problems and math word problems.Source: H. Full Results of MathVista · p. 13, Table 6 · p. 14
  • Figure 7 plots Pass@K and Reward-Pass@2 changes during M-STAR training across selected temperatures.Source: Figure 7 · p. 14

I. More Related Works

  • Self-evolving methods aim to improve reasoning without external supervision, while earlier approaches include single-round data augmentation and iterative retraining or continuous fine-tuning.Source: §I · p. 15
  • Multimodal reasoning work has used supervised fine-tuning, while newer self-evolving examples generate captions or missing explanations for retraining.Source: §I · p. 15
  • M-STAR uses answer filtering before process-reward reranking and avoids step-wise policy gradients; integrating process reward models into online RL remains a proposed direction.Source: §I · p. 15

Related papers

Loading 2412.17451v3…