PaperlayerView exact revision on arXiv

Source-linked AI summary

MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learning

Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, Ping Luo, Yu Qiao, Qiaosheng Zhang, Wenqi Shao

arXiv:2503.07365v2cs.CV

TL;DR

Prior multimodal reasoning research has faced limited dataset diversity, smaller model scales, and challenges in stable extended reinforcement learning. This paper introduces the MMK12 dataset and trains 7B and 32B MM-Eureka models using rule-based reinforcement learning. MM-Eureka-32B surpasses almost all open- and closed-source models on multidisciplinary multimodal reasoning, while scoring 1.7 points below o1 on MMK12’s multidisciplinary evaluation set.

  • Problem

    Prior multimodal reasoning work has used relatively small models and struggled with stable extended reinforcement learning, while high-quality multimodal mathematics datasets remain scarce.

    Source: §1 · p. 1

  • Method

    The paper constructs MMK12 and trains 7B and 32B MM-Eureka models with rule-based reinforcement learning, online prompt filtering, and a two-stage strategy for the 32B model.

    Source: §1 · pp. 2–3

  • Results

    Across multidisciplinary multimodal reasoning, MM-Eureka-32B surpasses almost all open- and closed-source models and scores 1.7 points below o1 on MMK12’s multidisciplinary evaluation set.

    Source: §1 · p. 3, §7 · p. 13

  • Takeaways & Limitations

    MM-Eureka-32B is a strong performer in multidisciplinary multimodal reasoning, though it remains slightly behind o1.

    Source: §7 · p. 13

    MM-Eureka-32B still trails closed-source models such as o1 on more challenging MathVision and OlympiadBench benchmarks, despite outperforming o1 on MathVista.

    Source: §5.4 · p. 9

Abstract

from arXiv · show

DeepSeek R1, and o1 have demonstrated powerful reasoning capabilities in the text domain through stable large-scale reinforcement learning. To enable broader applications, some works have attempted to transfer these capabilities to multimodal reasoning. However, these efforts have been limited by the limited difficulty of selected tasks and relatively small training scales, making it challenging to demonstrate strong multimodal reasoning abilities. To address this gap, we introduce the MMK12 dataset and MM-EUREKA with 7B and 32B parameters. The former is a high-quality multimodal mathematics reasoning dataset featuring diverse knowledge domains with human-verified answers and solution processes. The latter is a multimodal model employing rule-based reinforcement learning on MMK12, utilizing online filtering and two-stage training strategy to enhance training stability. MM-EUREKA demonstrates remarkable performance gains in multimodal mathematical reasoning, outperforming previous powerful models like InternVL2.5-78B or InternVL2.5-38B-MPO. In particular, MM-EUREKA achieves competitive or superior performance compared to both open-source and closed-source models, and trails slightly behind o1 in multidisciplinary reasoning tasks. We open-source our complete pipeline to foster further research in this area. We release all our codes, models, data, etc. at https://github.com/ModalMinds/MM-EUREKA

1 INTRODUCTION

Large-scale RL has advanced text reasoning, but its transfer to multimodal reasoning remains underexplored and stable training is unresolved. MM-Eureka addresses this gap with the MMK12 dataset and 7B/32B models trained for strong multimodal reasoning.

  • 72.3 on multidisciplinary K12 evaluation sets places MM-Eureka-32B above Qwen2.5-VL-72B (70.3) and second only to o1 (73.9).Source: §1 · p. 2
  • MM-Eureka-32B scores 73.4 on WeMath, outperforming all open-source models and most closed-source models, and 74.8 on MathVista, surpassing all open- and closed-source models.Source: §1 · p. 2
  • MMK12 spans functions, geometry, and equations from elementary through high school, with non-overlapping evaluation sets of 500 multiple-choice questions each in mathematics, physics, chemistry, and biology.K12 refers to primary and secondary education; the evaluation questions primarily cover secondary-level material.Source: §1 · pp. 2–3
  • Online filtering removes zero-advantage prompts, while 32B training uses MMK12 RL without KL followed by Geo3k fine-tuning with KL to support stable, long-horizon training.The approach uses GRPO for greater efficiency than PPO.Source: §1 · p. 3
  • The authors find RL mainly raises correct-answer generation probability, while math-only rule-based training transfers across disciplines and generalizes better than SFT and COT SFT.Source: §1 · p. 3
  • The authors open-source their models, code, and collected multimodal mathematical reasoning data, alongside support for a wider range of RL algorithms.Source: §1 · p. 3
  • MM-Eureka-7B scores 73.0 on MathVista, surpassing InternVL2.5-78B.Source: §1 · p. 3

2 RELATED WORK

Prior work developed text-reasoning methods and began exploring reinforcement learning for vision-language reasoning, but multimodal reasoning remains relatively underexplored and open-source efforts have important limitations.

2.1 LANGUAGE REASONING MODEL

2.2 VISION-LANGUAGE REASONING MODEL

  • GPT-4o (Hurst et al., 2024) and Kimi-VL (Team et al., 2025b) lead multimodal reasoning, while open-source models remain in early exploration.Recent work has started applying reinforcement learning to visual reasoning, but the paper identifies a need for a comprehensive open-source training pipeline.Source: §2.2 · p. 4
  • LMM-R1 relies mainly on text-only data for gains; R1-V targets limited subdomains; Reason-RFT (Tan et al., 2025) uses SFT before RL.The paper aims to provide an effective, stable, comprehensive open-source pipeline covering datasets, code, and models.Source: §2.2 · p. 4

3 MMK12: MULTIMODAL MATHEMATIC K12-LEVEL DATASET

MMK12 is a diverse K–12 multimodal mathematics dataset designed to address limited domain coverage and answer-correctness concerns in prior datasets, with verified answers and step-by-step solutions.

3 MMK12: MULTIMODAL MATHEMATIC K12-LEVEL DATASET

  • MMK12 contains over 15,000 multimodal mathematical reasoning problems across diverse domains, each with a standard reference answer and detailed step-by-step solution.The paper contrasts this breadth and answer quality with prior datasets’ narrower scope or uncertain answer correctness.Source: §3 · p. 4, Table 1 · p. 4
  • MMK12 problems are collected from Chinese mathematics textbooks and exams spanning elementary through high school, translated and refined into English with LLM help, and verified for reliable answers.The construction process retains only fill-in-the-blank training problems to minimize false positives and uses Math-Verify to parse answers; evaluation questions are selected as multiple choice.Source: §3 · pp. 4–5, Figure 2 · p. 5
  • The 15,616-item training set contains fill-in-the-blank problems, while the 2,000-item evaluation set uses multiple-choice questions spanning math, physics, chemistry, and biology.Training samples include a question, image, final answer, and CoT-formatted solution; evaluation items include a question, image, and correct answer choice.Source: §3 · pp. 4–5
  • The training set includes 455 elementary-, 9,776 middle-, and 5,385 high-school problems, with coverage including geometry, functions, spatial reasoning, and pattern reasoning.Function tasks involve graphs and expressions, geometric tasks apply spatial relationships and theorems, and pattern tasks use visual sequences and flow diagrams.Source: §3 · p. 5
  • Human involvement and verification support question diversity and correctness in answers and solutions, making MMK12 suitable for RL and SFT training and multidisciplinary model evaluation.The evaluation set is designed to provide a convenient and accurate test of multidisciplinary reasoning.Source: §3 · p. 5

4 METHOD

MM-Eureka trains 7B and 32B Qwen2.5-VL models with rule-based rewards and GRPO. Its method adds online prompt filtering and a two-stage strategy to address training instability and domain-specific performance gaps.

4.1 BASIC SETTINGS

  • The method uses Qwen2.5-VL 7B and 32B base models and a multimodal RL framework built on OpenRLHF, compatible with InternVL and QwenVL.Training uses GRPO, with rule-based format rewards of 0 or 0.5 and accuracy rewards of 0 or 1.Source: §4.1 · p. 5

4.2 REWARD FUNCTION

  • Following DeepSeek-R1 (DeepSeek-AI et al., 2025), the method uses sparse rule-based rewards rather than outcome or process reward models.Math-Verify assigns accuracy reward 1 or 0; format compliance with <think>...</think><answer>...</answer> earns 0.5 or 0.Source: §4.2 · p. 6

4.3 ADVANTAGE ESTIMATION AND POLICY UPDATE

  • GRPO (DeepSeek-AI et al., 2025) computes response advantages relative to other responses to each query and updates the policy with PPO-clip loss and a KL penalty.To mitigate loss spikes from large policy ratios combined with negative advantages, the method clips the ratio to [0, c], with default c = 3.Source: §4.3 · p. 6

4.4 ONLINE FILTERING

  • Online filtering stabilizes RL training: filtered models maintain accuracy reward and response length, while unfiltered models’ accuracy approaches zero and responses shorten.Prompts with responses that are all correct or all incorrect are filtered because their GRPO advantages are zero.Source: §4.4 · p. 6, Figure 3 · p. 7

4.5 TWO-STAGE TRAINING

  • For 32B training, policy-ratio spikes precede accuracy-reward collapse, and MMK12-only training degrades geometry performance; two-stage training addresses these issues and yields gains across nearly all benchmarks.Stage one applies GRPO without KL regularization on MMK12; stage two adds KL regularization and Geo3k data.Source: §4.5 · p. 7

5 EXPERIMENTS

Experiments compare MM-Eureka-7B and 32B with closed- and open-source models across multimodal math and MMK12 multidisciplinary benchmarks, reporting strong same-sized open-source performance and cross-discipline generalization.

5.1 EXPERIMENTS SETUP

  • MM-Eureka uses a DeepSeek-R1-style <answer>-tag prompt; 32B training disables KL in Stage 1 and adds a 1×10−3 KL penalty in Stage 2.Both rollout and training batch sizes are 128, with eight rollouts per sample and a learning rate of 1×10−6.Source: §5.1 · p. 8, Table 2 · p. 8

5.2 BASELINES

  • The comparison spans proprietary models, general-purpose open-source vision-language models, and open-source systems fine-tuned specifically for reasoning.The reasoning-focused baselines use approaches including SFT, DPO, and rule-based reinforcement learning.Source: §5.2 · p. 8

5.3 BENCHMARKS

  • Evaluation covers MathVista(testmini), MathVerse(testmini), MathVision(test), OlympiadBench(EN-OE), and WeMath, plus MMK12 for fundamental multimodal multidisciplinary problems common in K12 education, unlike OlympiadBench’s advanced graduate-level reasoning.Evaluation uses greedy decoding with temperature 0.Source: §5.3 · pp. 8–9

5.4 QUANTITATIVE RESULTS

  • 73.0 on MathVista lets MM-Eureka-7B exceed InternVL-78B by 0.7% and InternVL2.5-38B-MPO by 0.2%; 32B leads comparable open-source models except on MathVision.MM-Eureka-32B also approaches Qwen-72B-Instruct on average and outperforms Claude3.7 Sonnet on WeMath.Source: §5.4 · p. 9, Table 3 · p. 10

5.5 QUALITATIVE RESULTS

  • In physics, MM-Eureka-32B derives a 24N contact force using Newton’s second law, unlike Qwen2.5-VL-32B-Instruct’s incorrect 40N result; examples show similar reasoning gains across subjects.The qualitative comparisons cover mathematics, physics, chemistry, and biology, and describe improved multi-step deduction over the base model.Source: §5.5 · pp. 10–11

5.4 QUANTITATIVE RESULTS

  • On MMK12, MM-Eureka-32B trails o1 by 1.7% while outperforming larger models; MM-Eureka-7B improves chemistry by 9.8% and biology by 11.2%.MM-Eureka-7B also exceeds InternVL2.5-VL-78B overall, while MM-Eureka-32B still trails o1 on MathVision and OlympiadBench; o1 results use a random 500-instance sample.Source: §5.4 · pp. 9–10, Table 4 · p. 11

6 DISCUSSION

The discussion argues that RL strengthens reasoning and generalizes beyond mathematics, while staged training addresses instability and domain-specific capability gaps. Its results support separating reasoning improvements from knowledge acquisition, though the authors identify broader-domain reasoning transfer as future work.

6.1 ARE KNOWLEDGE AND REASONING DECOUPLED?

  • Across eight responses, MM-Eureka improved accuracy on problems Qwen had sometimes solved, but not on problems it had never solved correctly.The authors interpret this pattern as reasoning better exploiting available knowledge, while reasoning alone does not supply missing knowledge.Source: §6.1 · p. 11, Figure 4 · p. 12
  • The authors’ results support separating knowledge learning from reasoning learning, and they propose generalizing mathematics-trained reasoning to broader domains as future work.They note that mathematics offers clear logic, structured responses, and verifiable answers; despite math-focused training, the model also improved on physics, chemistry, and biology questions.Source: §6.1 · pp. 11–12

6.2 RL GENERALIZES BETTER THAN SFT

  • RL outgeneralizes SFT and COT SFT, raising mathematics and physics scores by 12.8 and 10.8 points, respectively.The reported generalization gains are particularly notable on out-of-distribution Physics, Chemistry, and Biology tests.Source: §6.2 · p. 12, Table 5 · p. 12

6.3 TWO-STAGE TRAINING

  • Without KL divergence, continued RL training often collapsed as policy-ratio instability increased; adding KL divergence in stage two stabilized learning.The two-stage strategy uses K12 data first and Geo3k data second, omitting KL divergence in stage one and adding it in stage two.Source: §6.3 · pp. 12–13, Figure 5 · p. 13
  • MathVista geometry problem solving fell from 74.0 to 56.7 after stage one, while Geo3k stage-two training improved multiple mathematical benchmarks.The authors describe the second phase as addressing performance gaps left by the first stage.Source: §6.3 · p. 13, Table 6 · p. 13

7 CONCLUSION

MM-Eureka introduces MMK12 and a more stable, efficient rule-based reinforcement learning approach, producing 7B and 32B models that perform strongly on multimodal reasoning tasks.

7 CONCLUSION

  • MM-Eureka-7B and MM-Eureka-32B are top performers among similarly sized multimodal reasoning models; MM-Eureka-32B surpasses almost all open- and closed-source models on multidisciplinary reasoning, trailing only slightly behind o1.Source: §7 · p. 13
  • MMK12 is a high-quality multimodal mathematical reasoning dataset collected by the authors.Source: §7 · p. 13
  • The training approach combines online filtering with a two-stage strategy to make rule-based reinforcement learning more stable and efficient.Source: §7 · p. 13

A APPENDIX: QUALITATIVE ANALYSIS

The appendix presents qualitative examples in mathematics, physics, chemistry, and biology, contrasting reasoning errors by Qwen2.5-VL-32B-Instruct with correct reasoning by MM-Eureka-32B.

  • In mathematics, the example marks Qwen2.5-VL-32B-Instruct’s reasoning errors and MM-Eureka-32B’s correct reasoning.Source: Figure 6 · p. 18
  • In physics, the example contrasts Qwen2.5-VL-32B-Instruct’s reasoning errors with MM-Eureka-32B’s correct reasoning.Source: Figure 7 · p. 19
  • In chemistry, the example contrasts Qwen2.5-VL-32B-Instruct’s reasoning errors with MM-Eureka-32B’s correct reasoning.Source: Figure 8 · p. 20
  • In biology, the example contrasts Qwen2.5-VL-32B-Instruct’s reasoning errors with MM-Eureka-32B’s correct reasoning.Source: Figure 9 · p. 21

MM-EUREKA

The passages describe titration of diprotic oxalic acid with NaOH through two equivalence points, with the solution’s composition and pH changing at each volume.

  • H2C2O4 dissociates in two steps, and strong-base NaOH neutralizes its H+ ions step by step.Source: MM-EUREKA · p. 22
  • At 25 mL NaOH, the first equivalence point converts all H2C2O4 to NaHC2O4, with pH around 2.Source: MM-EUREKA · p. 22
  • At 40 mL NaOH, the solution contains NaHC2O4 and Na2C2O4, and is slightly acidic at pH 5.5.The passage attributes the acidity to HC2O4^- dissociation exceeding C2O4^2- hydrolysis.Source: MM-EUREKA · p. 22
  • At 50 mL NaOH, the second equivalence point converts all H2C2O4 to Na2C2O4, and the pH is 8.7.The supplied solution identifies answer C.Source: MM-EUREKA · p. 22

Related papers

Loading 2503.07365v2…