PaperlayerView exact revision on arXiv

Source-linked AI summary

MathSE: Improving Multimodal Mathematical Reasoning via Self-Evolving Iterative Reflection and Reward-Guided Fine-Tuning

Jinhao Chen, Zhen Yang, Jianxin Shi, Tianyu Wo, Jie Tang

arXiv:2511.06805v1cs.AIcs.LG

TL;DR

Static teacher-derived math datasets provide limited adaptation to novel problems and lack iterative reasoning depth. MathSE iteratively fine-tunes MLLMs using correct generated reasoning paths and ORM feedback; it improves performance across multimodal math benchmarks and outperforms QVQ on MathVL-test.

  • Problem

    Prior math fine-tuning datasets are often static and teacher-derived, limiting adaptation to novel or intricate problems and providing insufficient iterative depth for robust generalization.

    Source: Introduction · p. 1

  • Method

    MathSE iteratively fine-tunes MLLMs on correct reasoning paths generated by the model and uses an ORM to assess correctness and analyze errors.

    Source: Introduction · p. 2, Specialized Outcome Reward Model (ORM) · p. 4

  • Results

    Average score increases were 15.91% for CogVLM2, 12.28% for Qwen2-VL-7B, and 8.04% for InternVL2.5-8B across MathVista, MathVerse, and MathVision, and MathSE outperformed QVQ on MathVL-test.

    Source: Introduction · p. 2, Main Results · p. 6

  • Takeaways & Limitations

    The experiments support MathSE as an effective approach for improving MLLM performance on challenging multimodal mathematical reasoning benchmarks.

    Source: Conclusion · p. 8

Abstract

from arXiv · show

Multimodal large language models (MLLMs) have demonstrated remarkable capabilities in vision-language answering tasks. Despite their strengths, these models often encounter challenges in achieving complex reasoning tasks such as mathematical problem-solving. Previous works have focused on fine-tuning on specialized mathematical datasets. However, these datasets are typically distilled directly from teacher models, which capture only static reasoning patterns and leaving substantial gaps compared to student models. This reliance on fixed teacher-derived datasets not only restricts the model's ability to adapt to novel or more intricate questions that extend beyond the confines of the training data, but also lacks the iterative depth needed for robust generalization. To overcome these limitations, we propose \textbf{\method}, a \textbf{Math}ematical \textbf{S}elf-\textbf{E}volving framework for MLLMs. In contrast to traditional one-shot fine-tuning paradigms, \method iteratively refines the model through cycles of inference, reflection, and reward-based feedback. Specifically, we leverage iterative fine-tuning by incorporating correct reasoning paths derived from previous-stage inference and integrating reflections from a specialized Outcome Reward Model (ORM). To verify the effectiveness of \method, we evaluate it on a suite of challenging benchmarks, demonstrating significant performance gains over backbone models. Notably, our experimental results on MathVL-test surpass the leading open-source multimodal mathematical reasoning model QVQ. Our code and models are available at \texttt{https://zheny2751\allowbreak-dotcom.github.io/\allowbreak MathSE.github.io/}.

Introduction

MathSE addresses the limits of static teacher-distilled reasoning data with iterative, reward-guided self-evolution for multimodal mathematical reasoning. The introduction reports gains across multimodal math benchmarks and describes a cycle of inference, reflection, and fine-tuning.

Introduction

  • Teacher-distilled datasets capture static reasoning patterns that can limit adaptation to novel or intricate problems and lack the iterative depth needed for robust generalization.The introduction also notes a gap between static teacher-derived patterns and student-model data distributions.Source: Introduction · p. 1
  • MathSE first fine-tunes on a subset of GPT-4o-distilled CoT data, then uses correct reasoning paths generated on the remaining dataset for further fine-tuning.The iterative cycle learns from the model’s previous inferences.Source: Introduction · p. 2
  • An Outcome Reward Model evaluates entire reasoning processes, identifies erroneous steps, and provides analyses that guide reflection and refinement of incorrect paths.GPT-4o uses these ORM-provided errors and analyses to produce corrected paths for a reflection dataset used to train the final model.Source: Introduction · p. 2
  • As self-evolving training progresses, more examples are classified correctly, improving overall accuracy and progressively enhancing problem-solving skills.The framework is presented as bridging static teacher-derived data and dynamic student learning.Source: Introduction · p. 2
  • MathSE improves performance across MathVista, MathVL-test, MathVerse, and Math-Vision; its approximately 10B models outperform similar-sized peers, match Claude 3.5 Sonnet levels, and surpass QVQ on MathVL-test.Experiments use CogVLM2, Qwen2-VL-7B, and InternVL2.5-8B backbones.Source: Introduction · p. 2

Related Work

Related work identifies limits in multimodal mathematical reasoning, static fine-tuning and distillation, and existing reward and reflection approaches. MathSE addresses these gaps with iterative refinement and diagnostic ORM-guided reflection.

Multimodal Math Reasoning

  • MLLMs improve visual reasoning over caption-based text models, yet still struggle with complex math requiring deep logic, precise visual interpretation, and multistep solving.Source: Multimodal Math Reasoning · pp. 2–3

Supervised Fine-Tuning and Knowledge Distillation

  • SFT adapts models using labeled multimodal math data but requires costly high-quality labels; KD scales teacher-synthesized QA yet may neglect in-distribution data, limiting student generalization.MathSE instead emphasizes in-distribution data and iterative refinement over static fine-tuning.Source: Supervised Fine-Tuning and Knowledge… · p. 3

Reward Models for Reasoning Tasks

  • MathSE’s ORM uses diagnostic error analysis to identify critical reasoning errors for targeted revision, avoiding exhaustive intermediate correctness labels required by process supervision.Traditional ORMs assign outcome-quality scores without assessing reasoning processes, so they provide no actionable feedback for improving intermediate steps.Source: Reward Models for Reasoning Tasks · p. 3

Self-Improvement and Reflection Mechanism

  • Prior reflection methods let models reconsider incorrect outputs but are primarily text-based; MathSE combines reflection with ORM-guided, error-specific feedback for iterative multimodal math improvement.Prior self-improvement work includes Qwen2.5-Math’s pipeline-wide mechanisms for improving accuracy, reducing errors, and generalizing across mathematical tasks.Source: Self-Improvement and Reflection… · p. 3

Methodology

MathSE iteratively improves multimodal mathematical reasoning by combining GPT-4o-distilled supervised fine-tuning, ORM feedback, and reflection on incorrect solutions.

Framework Overview

  • The framework fine-tunes a base model on GPT-4o-distilled data, evaluates new reasoning paths with an ORM, and uses reflections on errors to improve the final model.Figure 3 presents the workflow as a three-stage iterative process.Source: Framework Overview · p. 3, Figure 3 · p. 4

Supervised Fine-Tuning with GPT-4o Distilled Data

Specialized Outcome Reward Model (ORM)

  • The ORM judges each reasoning path as correct or incorrect and, for errors, identifies the faulty step and explains the reasoning flaw.It is fine-tuned on a balanced 60k-sample dataset: 30k annotated incorrect paths and 30k correct Chain-of-Thought examples.Source: Specialized Outcome Reward Model (ORM) · p. 4

Reflection and Self-Improvement

  • GPT-4o uses the ORM’s error steps and analyses to reflect on incorrect paths and generate corrected solutions.The reflection prompt supplies the previous solution, faulty step, and error analysis, then asks for a corrected solution.Source: Reflection and Self-Improvement · p. 4

Iterative Training Process

  • Across a set number of rounds, the model evaluates paths on remaining data, adds correct and reflected paths to training data, and fine-tunes an updated model.The remaining dataset excludes covered paths; the updated training set is used for subsequent fine-tuning, with reflected paths added to the final training set.Source: Iterative Training Process · pp. 4–5

Experiments · Experimental Setup · Main Results

MathSE is evaluated across multimodal math benchmarks and backbones, with reported gains on MathVL-test and three public benchmarks. The experiments use regenerated solutions from a diverse dataset spanning school-level mathematical domains.

Experiments

  • The study evaluates MathSE’s effectiveness and generalization across multiple standard benchmarks.Source: Experiments · p. 5

Dataset

  • MathSE uses MathVL’s 341,346 Chinese K12 multimodal exercises alongside open-source datasets, spanning arithmetic, algebra, geometry, probability, and applied word problems.Source: Dataset · p. 5
  • Regenerated solutions extended average answer length from 325 to 792 characters, providing more complete procedures and structured reasoning pathways.Source: Dataset · p. 5

Public Benchmarks

  • Evaluation covers MathVista, MathVerse, MathVision, and the specially constructed MathVL-test dataset.Source: Public Benchmarks · p. 5

Baselines

  • Comparisons include closed-source and open-source MLLMs described as strong baselines representing the current state of the art in multimodal reasoning.Source: Baselines · p. 5

Implementation Details

Main Results

  • 65% accuracy was the highest MathVL-test result, achieved by MathSE-InternVL across the three backbones.The passage attributes substantial improvements across all three models to iterative training.Source: Table 1 · p. 6, Main Results · p. 6
  • Average scores rose 15.91% (CogVLM2), 12.28% (Qwen2-VL-7B), and 8.04% (InternVL2.5-8B); MathVista (GPS) gains were 31.06%, 25.96%, and 15.39%, respectively.The reported MathVista gains are for CogVLM2, Qwen2-VL-7B, and InternVL2.5-8B, respectively.Source: Main Results · p. 6

Ablation Studies · Error Correction Analysis

The ablations favor ORM feedback, iterative self-evolving training data, and fine-grained ORM error analysis, while transition counts show MathSE increasingly retains correct predictions and reduces persistent or introduced errors.

Ablation Studies

  • All ablation experiments used CogVLM2 as the backbone model for consistent evaluation.Source: Ablation Studies · p. 6
  • 64.70% accuracy with ORM feedback exceeded GPT-4o feedback (64.25%) and no reflection (62.35%), supporting ORM feedback and retaining reflection.The reflection ablation compares ORM feedback, GPT-4o feedback, and no reflection.Source: Ablation Studies · p. 6
  • 62.35% accuracy for the proposed data mix exceeded 58.00% for full GPT-4o on MathVL-test.The proposed 240k paths combine 90k GPT-4o-generated paths with 150k iteratively generated, filtered, and refined paths; full GPT-4o uses 240k GPT-4o-generated paths filtered by ORM.Source: Table 4 · p. 6, Ablation Studies · p. 6

Error Correction Analysis

  • Correct→correct samples rose from 402 initially to 1018 by the final stage, demonstrating increased retention of correct predictions.The analysis tracks transitions between stages of MathSE.Source: Error Correction Analysis · p. 7
  • Persistently incorrect and correct→incorrect examples both decreased across transitions, reinforcing greater prediction stability.The authors attribute progressive accuracy improvement to leveraging past mistakes and refining learned representations.Source: Error Correction Analysis · p. 7

Ablation Studies

  • On ORM-2K, the proposed ORM achieved higher accuracy than binary-only feedback for both correct and incorrect reasoning paths.ORM-2K is a balanced test set of 1,000 correct and 1,000 incorrect paths; the proposed ORM adds error-step identification and analysis to binary judgments.Source: Ablation Studies · pp. 6–7, Table 5 · p. 7

Conclusion

MathSE improves MLLM reasoning through iterative fine-tuning, reward-guided feedback, and reflection. Its specialized outcome reward model and reflection mechanism support progressive reasoning improvement, with substantial gains across challenging multimodal mathematical reasoning benchmarks.

Conclusion

  • MathSE combines iterative fine-tuning, reward-guided feedback, and reflection to enhance MLLM reasoning capabilities.The framework addresses limitations of traditional one-shot fine-tuning.Source: Conclusion · p. 7
  • MathSE achieves substantial performance improvements across challenging multimodal mathematical reasoning benchmarks.The conclusion attributes these gains to the framework’s iterative fine-tuning, reward-guided feedback, and reflection.Source: Conclusion · pp. 7–8
  • A specialized outcome reward model and reflection mechanism enable progressive improvement in reasoning ability.The conclusion presents these mechanisms as addressing limitations of traditional reward models.Source: Conclusion · p. 8

SFT Configuration

The SFT setup shares an AdamW optimizer with cosine decay across models, while training steps, input resolutions, and warmup ratios vary by architecture. All models use a global batch size of 128 and weight decay of 5×10−2.

SFT Configuration

  • All models use AdamW with cosine decay learning-rate scheduling during SFT.The configurations are summarized in Table 6.Source: SFT Configuration · p. 11
  • Training steps and input resolutions vary across architectures, with CogVLM2 using higher resolution and Qwen2-VL-7B using dynamic resolution.These differences reflect model-specific design philosophies and computational characteristics.Source: SFT Configuration · p. 11
  • All models are trained with a global batch size of 128 and weight decay of 5×10−2.Warmup ratios differ slightly among models, reflecting differing needs for early learning-rate ramp-up.Source: SFT Configuration · p. 11

Error Analysis

On MathVL-test, reasoning errors are the largest source of mistakes across the evaluated models, followed by question misunderstanding; the analysis recommends prioritizing both areas for improvement.

Error Analysis

  • Reasoning errors dominate mistakes at 63.1% for MathSE-Qwen, 62.9% for MathSE-CogVLM2, and 65.3% for MathSE-InternVL.The analysis identifies complex reasoning as a significant challenge in multimodal understanding.Source: Error Analysis · p. 11
  • Question misunderstanding is the second-largest error category at 20.8% for MathSE-Qwen, 21.6% for MathSE-CogVLM2, and 27.0% for MathSE-InternVL.The analysis suggests that models sometimes struggle to interpret user intent.Source: Error Analysis · p. 11
  • Knowledge errors account for 9.4% for MathSE-CogVLM2, 7.7% for MathSE-Qwen, and 6.3% for MathSE-InternVL, while calculation errors are 1.7%, 1.1%, and 1.0%, respectively.Knowledge errors are the third most common type, while calculation errors are relatively infrequent, suggesting basic arithmetic is not a major hurdle.Source: Error Analysis · p. 11
  • Vision recognition errors vary across models: MathSE-CogVLM2 has 4.1%, compared with 0.6% for MathSE-Qwen and 0.4% for MathSE-InternVL.The analysis suggests these differences may reflect variation in the models’ vision processing capabilities.Source: Error Analysis · p. 11

Prompts for Data Generation

The section presents prompts for generating image-grounded reasoning paths and for evaluating incorrect answers and reflecting to produce improved solutions.

Reasoning Path Generation Prompt

  • The reasoning-generation prompt requests a detailed, step-by-step solution based on the question and image, ending with a standardized final-answer statement.Source: Prompts for Data Generation · p. 11

Feedback Generation Prompt

  • The feedback prompt compares a predicted answer with the standard answer, checks reasoning and error causes, and requires objective, evidence-based improvement suggestions.It treats equivalent answer formats and any of multiple valid answers as correct, and asks for suggestions even when the answer is correct.Source: Prompts for Data Generation · pp. 11–12
  • The reflection prompt revisits a confirmed-wrong answer using the question, image, and possibly unreliable error feedback, then requests a detailed reconsidered solution and final-answer statement.Source: Prompts for Data Generation · p. 12

Training Data Examples

MathSE uses ORM-verified reasoning examples for training and reflection, incorporating correct paths into the corpus and using feedback to identify and correct errors.

Correct Reasoning Path Example

  • ORM-verified correct reasoning paths are added to the training corpus to enhance Math-Vision’s reasoning capabilities.Figure 5 illustrates an example of a validated correct path.Source: Correct Reasoning Path Example · p. 12

Refined Reasoning Path with ORM Feedback

Error Cases

The section illustrates five categories of reasoning errors identified by the ORM and shows how its feedback diagnoses a concrete calculation mistake.

  • The ORM identifies reasoning, knowledge, calculation, question-understanding, and vision-recognition errors in the examples shown in Figures 7–11.Source: Error Cases · p. 12, Figure 7 · p. 15, Figure 8 · p. 16
  • In a cylinder-and-cone example, the ORM flags an incorrect answer caused by treating each 2 dm diameter as a 2 dm radius.The predicted total was 62.8 cubic decimeters; the ORM feedback says the radius should be 1 dm and labels the error as incorrect radius use and calculation.Source: Error Cases · p. 14, Figure 6 · p. 14
Loading 2511.06805v1…