Source-linked AI summary
SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning
Zhongwei Wan, Zhihao Dou, Che Liu, Yu Zhang, Dongfei Cui, Qinjian Zhao, Hui Shen, Jing Xiong, Yi Xin, Yifan Jiang, Chaofan Tao, Yangfan He, Mi Zhang, Shen Yan
TL;DR
MLLMs struggle with complex reasoning that requires explicit self-reflection and self-correction, while pre-training limits the reasoning and knowledge improvements obtainable from existing methods. SRPO addresses this gap with a two-stage reflection-aware training framework combining reflection-focused SFT and GRPO-based rewards. Across multimodal reasoning benchmarks, it surpasses existing state-of-the-art models in reasoning accuracy and reflection quality.
Problem
MLLMs struggle with explicit self-reflection and self-correction, while pre-training constrains the reasoning capabilities that existing approaches can improve.
Method
SRPO combines advanced-MLLM-generated reflection data for multimodal SFT with a tailored GRPO method using rewards for meaningful reflective reasoning.
Results
SRPO surpasses existing state-of-the-art models across multimodal reasoning benchmarks in reasoning accuracy and reflection quality.
Takeaways & Limitations
The results highlight reflection-driven training as an effective strategy for robust multimodal reasoning.
Takeaways & Limitations
Experiments cover dense MLLMs at 7B and 32B scales and publicly available multimodal reasoning datasets, without testing MoE or diffusion architectures or larger commercial datasets.
Abstract
from arXiv · showhide
Multimodal large language models (MLLMs) have shown promising capabilities in reasoning tasks, yet still struggle with complex problems requiring explicit self-reflection and self-correction, especially compared to their unimodal text-based counterparts. Existing reflection methods are simplistic and struggle to generate meaningful and instructive feedback, as the reasoning ability and knowledge limits of pre-trained models are largely fixed during initial training. To overcome these challenges, we propose Multimodal Self-Reflection enhanced reasoning with Group Relative Policy Optimization (SRPO), a two-stage reflection-aware reinforcement learning (RL) framework explicitly designed to enhance multimodal LLM reasoning. In the first stage, we construct a high-quality, reflection-focused dataset under the guidance of an advanced MLLM, which generates reflections based on initial responses to help the policy model learn both reasoning and self-reflection. In the second stage, we introduce a novel reward mechanism within the GRPO framework that encourages concise and cognitively meaningful reflection while avoiding redundancy. Extensive experiments across multiple multimodal reasoning benchmarks, including MathVista, MathVision, MathVerse, and MMMU-Pro, using Qwen-2.5-VL-7B and Qwen-2.5-VL-32B demonstrate that SRPO significantly outperforms state-of-the-art models, achieving notable improvements in both reasoning accuracy and reflection quality.
1 Introduction
SRPO addresses multimodal reasoning failures caused by redundant or erroneous steps and limits established during pre-training by combining reflection-focused SFT with reflection-aware reinforcement learning. Across multimodal benchmarks, it improves reasoning accuracy, reflection quality, and cross-task generalization.
- Motivation: Existing multimodal reasoning methods often produce redundant, repetitive, or erroneous steps because generation relies on local token-level dependencies.These issues can prevent reasoning models from substantially outperforming fast-thinking models.
- Motivation: Pre-training constrains the reasoning capabilities of MLLMs, limiting the effectiveness of prompting and reinforcement learning alone.The paper motivates external reflective experiences and cognitively guided techniques to surpass these boundaries.
- Approach: The framework integrates explicit self-reflection and iterative self-correction into both SFT and RL rather than relying only on extended chain-of-thought supervision.Its goal is to improve reasoning performance while also fostering self-reflection.
- Approach: SRPO is a two-stage framework that uses advanced-MLLM-generated reflections for multimodal SFT before reflection-aware reinforcement learning.The first stage generates reflections from discrepancies between policy outputs and ground truth, then uses them for cold-start initialization.
- Results: SRPO consistently outperforms current state-of-the-art models in reasoning accuracy, reflection quality, and cross-task generalization.The paper reports extensive evaluations across multimodal reasoning benchmarks using representative Qwen-2.5-VL models.
- Contributions: The method combines a reflection-focused data-generation pipeline with a GRPO-based reward function that incentivizes meaningful reflective reasoning.These are presented as the paper’s principal contributions to multimodal reasoning training.
2 Related Works
Prior work applies reinforcement learning and self-reflection to improve language and multimodal reasoning, but existing multimodal approaches remain limited in their ability to close the capability gap.
- Reinforcement Learning for LLM Reasoning: Large-scale reinforcement learning has improved complex language-model reasoning through result-oriented or formatting-specific reward signals.Several studies directly fine-tune base language models with reinforcement learning without supplementary supervised fine-tuning.
- Reinforcement Learning for Multimodal LLM Reasoning: Recent multimodal reinforcement-learning studies target open-source models, but some primarily benefit textual scenarios or rely on limited two-stage, rule-based training.The related work positions multimodal RL as an emerging response to proprietary-model dominance.
3 Method of SRPO
SRPO combines reflection-oriented supervised fine-tuning with reflection-aware GRPO to teach multimodal models to identify, revise, and improve reasoning while limiting redundancy. Its two-stage pipeline uses generated reflective examples for cold-start initialization and rewards reflection according to formatting, brevity, and its effect on answer correctness.
- Reflection-oriented SFT: SRPO first constructs reflection data from initial policy responses, using larger MLLMs to refine correct reasoning or revise incorrect reasoning before supervised fine-tuning.Each example contains an initial response, generated reflection, and ground-truth answer, transferring reflective reasoning into the policy model.
- Reflection-oriented SFT: The cold-start stage trains the policy model to use reflection to revise its initial response toward the ground-truth answer and align with correct reasoning trajectories.The process is intended to provide self-reflective reasoning capabilities before reinforcement learning.
- Reflection-aware RL: SRPO uses GRPO to compare grouped responses through reward-based policy optimization and intra-group advantage estimation without a critic model.The objective includes clipping and a KL-divergence penalty, while response advantages are computed from sampled-group rewards.
- Reflection-aware RL: The total reward combines task and reflection rewards for responses structured as first solution, reflection, and second refined solution.The reflection component includes formatting, target-length brevity, and effectiveness based on whether reflection preserves or improves correctness.
- Reflection-aware RL: The task reward combines format supervision with answer accuracy, encouraging structured reasoning and correct first solutions.The format reward checks <think>...</think> boundaries, while the accuracy reward checks agreement with the ground truth.
- Reflection-aware RL: The effectiveness reward assigns +0.25 for preserving a correct answer, +0.5 for correcting an incorrect answer, 0 for failing to correct it, and −0.25 for making a correct answer wrong.The brevity reward peaks at a target length and smoothly decays toward zero near the maximum, encouraging concise, informative reflection.
4 Experiment
SRPO is evaluated across multimodal general, mathematical, and cross-disciplinary reasoning benchmarks, with ablations examining data scale, reflection components, qualitative behavior, and training dynamics. Across these evaluations, reflection-enhanced training improves benchmark performance, generalization, and the quality of reasoning revisions.
- Experiment Settings: The evaluation compares SRPO models with closed-source, open-source general, and open-source reasoning MLLM baselines across multimodal benchmarks.The experiments use Qwen2.5-VL-7B and Qwen2.5-VL-32B, with results reported against multiple baseline groups.
- Multimodal General Reasoning: SRPO-7B consistently outperforms existing open-source MLLMs on MMMU-Pro, MMMU, and EMMA, while SRPO-32B exceeds Gemini2-flash by 4.6 on EMMA.These results support generalization beyond mathematical reasoning tasks.
- Multimodal Mathematical Reasoning: SRPO achieves competitive mathematical reasoning performance, including 78.5% on MathVista and advantages over open-source reasoning models on OlympiadBench.The reported results attribute the gains to explicitly enhancing self-reflection during both SFT and RL.
- Cross-disciplinary Reasoning: SRPO surpasses MM-Eureka-7B by 5.1 points on Physics and OpenVLThinker-7B by 9.7 points on Chemistry in cross-disciplinary evaluation.The evaluation covers novel physics, chemistry, and biology tasks not included in training.
- RL Train-Set Size: With 5K RL samples, SRPO exceeds standard GRPO by 7.1 points on Physics, while performance consistently improves with more training data.This indicates that reflection-enhanced RL remains effective under reduced data scale.
- Effectiveness of Self-Reflection: Removing Self-Reflection RL leaves only minimal gains over Qwen-2.5-VL-7B, whereas removing Self-Reflection SFT still preserves a 5.1-point advantage over standard GRPO on Physics.The ablation identifies explicit reflection incentives during RL as essential, while both SFT and RL contribute to performance.
- Reasoning Qualitative Analysis: SRPO-generated reasoning explicitly revises erroneous steps and produces concise corrections, unlike typical GRPO outputs that retain flawed reasoning paths.Qualitative examples show reflection followed by revised reasoning and refined final answers during both training and testing.
- RL Training Dynamics Analysis: SRPO and SRPO without self-reflection SFT converge faster and outperform standard GRPO, while reflection-enhanced variants also improve PPO and DAPO.The cross-algorithm results indicate that self-reflection integration improves training behavior beyond the SRPO configuration.
5 Conclusion
SRPO is a reflection-aware reinforcement learning framework for enhancing multimodal reasoning. Experiments across multiple benchmarks show improvements in reasoning accuracy and reflection quality.
- SRPO enhances multimodal reasoning through reflection-aware reinforcement learning.The framework targets both reasoning capability and self-reflective behavior.
- SRPO significantly outperforms existing state-of-the-art models in reasoning accuracy and reflection quality.
- Reflection-driven training strategies play a critical role in robust multimodal reasoning.
7 Contributions
The listed contributors are affiliated with The Ohio State University, Case Western Reserve University, the University of Southern California, and The University of Hong Kong, with one contributor's work completed during an internship at ByteDance Seed.
- Zhongwei Wan is affiliated with The Ohio State University.
- Zhihao Dou is affiliated with Case Western Reserve University.
- One contributor's work was completed during an internship at ByteDance Seed.
- The listed affiliations also include the University of Southern California and The University of Hong Kong.
A.1 Limitation
The evaluation is limited to dense MLLMs at 7B and 32B scales and publicly available multimodal reasoning data, while the paper also discusses risks from misleading or biased training data and restricted access.
- Limitations: SRPO was evaluated on dense MLLMs at 7B and 32B scales, without scaling experiments on MoE or diffusion LM architectures.
- Limitations: The reinforcement learning data came exclusively from publicly available multimodal reasoning datasets.
- Future Work: Future work includes extending SRPO to MoE models and larger-scale reinforcement learning datasets.
- Social Impacts: Reflecting on misleading data may produce more convincing yet inaccurate or biased content.
- Social Impacts: Careful dataset curation, bias mitigation, and safeguards are presented as important for responsible deployment.
B.1 Further Experiment
Further experiments examine reward-coefficient sensitivity, reflection quality through human and LLM-based evaluation, reasoning-type distributions, and training efficiency.
- Reward Sensitivity: α = 0.05 performs similarly to α = 0.1, while α = 0.3 slightly degrades performance.The sensitivity experiments varied only the reflection-length reward coefficient under otherwise identical settings.
- Reflection Quality: Human experts judged 70% of reflections effective, and reflection corrected 13 of 33 initially incorrect answers.The evaluation sampled 100 MathVista test questions.
- Reflection Quality: The wrong-answer fix rate after reflection was 39%.
- Reflection Quality: GPT-4o scored reflections on logical flaws, missing assumptions, clarity, and actionable suggestions.
- Training Analysis: Figure 6 reports reasoning-type distributions across different training stages.
- Efficiency: SRPO requires modest additional wall-clock time compared with GRPO while delivering substantially better reasoning quality.Training efficiency was reported for 7B and 32B settings, with inference efficiency measured on the MathVista test-mini split.
B.2 Training Dataset
SRPO builds reflection-aware training data from multimodal reasoning sources and uses generated critiques, revisions, and structured prompts to train self-reflection and self-correction.
- Self-reflection SFT Dataset: The self-reflection SFT dataset combines multimodal reasoning sources spanning charts, diagrams, mathematics, physics, science, and general visual reasoning.The source collection includes ChartQA, DVQA, AI2D, MATH, Virgo, R1-OneVision, MMK12, PhyX, and other multimodal reasoning datasets.
- Self-reflection SFT Dataset: GPT-o4-mini converts selected reasoning samples into concise self-reflective feedback and structures them with the proposed self-reflection SFT template.The feedback is generated from the reasoning steps and associated ground-truth answers.
- Self-reflection RL Dataset: The RL dataset samples multimodal benchmarks with distinct reasoning characteristics, including ScienceQA and geometry-focused GeoQA.ScienceQA emphasizes visual, textual, and multiple-choice science reasoning, while GeoQA targets numerical reasoning over geometry.
- Reflection Prompt Design: The reflection prompts require an initial reasoning-and-answer pass, a critique of flaws and missing information, and a final revised reasoning-and-answer pass.The RL template also specifies structured tags and answer-format constraints.
- Reflection Prompt Design: The RL prompt requires the second reasoning process to incorporate improvements from reflection, even when the final answer remains unchanged.This explicitly trains revision rather than merely producing a second answer.
B.5 Generated Samples Analysis
Generated examples show SRPO revising incorrect reasoning through targeted reflection and simplifying correct reasoning while preserving the answer.
- SRPO Samples: SRPO examples show Qwen-2.5-VL-7B detecting errors and revising reasoning in both training samples and downstream evaluation cases.The examples include corrections of visual-task classification, geometry, chemistry, and angle reasoning.
- SRPO Samples: An angle example rejects an unsupported parallel-line assumption and revises the answer from 57 to 123 degrees after reconsidering the geometric relationship.The reflection explicitly flags the missing specification of parallel lines before producing the revised answer.
- SRPO Samples: A robotic-arm example changes the answer from incorrect card sorting to object recognition after reflection identifies 3D-perception evidence and the absence of sorting mechanisms.The revised reasoning links the camera, cards, and point-cloud patterns to object recognition.
- SRPO Samples: A perimeter-cost example corrects an overestimated expression by excluding internal construction segments and tracing only the external boundary.The revised perimeter is 10 + (x + y), with x + y = 11 and a final cost of 777.
- SRPO Samples: A chemistry example simplifies a correct solution into three steps while retaining answer A as the only chemically incorrect statement.The revision identifies the period, maps the elements, and eliminates the options more directly.
B.6 Training Dynamics
Training curves indicate that SRPO produces longer reflective responses, higher accuracy rewards, and stable policy updates compared with the reported baselines.
- Response and Reward Dynamics: SRPO consistently generates longer total responses and shows greater response-length growth than GRPO during training.The paper attributes this pattern to explicit emphasis on self-reflection.
- Response and Reward Dynamics: SRPO consistently achieves higher accuracy-reward values than the baselines during training.The authors interpret this as evidence that reinforcing reflective reasoning improves reasoning capabilities.
- Policy Stability: SRPO maintains ratio-clip-lower values below 0.005 whether self-reflection is used in both SFT and RL or only in RL.The curves are presented as indicating stable policy updates with moderate gradient adjustments.
- Policy Stability: Figure 7 compares training dynamics for SRPO, GRPO, and SRPO without self-reflection SFT.The comparison isolates the contribution of self-reflection SFT within the reported training dynamics.