Source-linked AI summary
LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RL
Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, Xu Yang
TL;DR
Compact 3B LMMs struggle with multimodal reasoning because visual tasks combine perception and logic, while answers can be ambiguous and complex reasoning data are scarce. LMM-R1 addresses this with text-only Foundational Reasoning Enhancement followed by Multimodal Generalization Training, achieving broad benchmark gains and improved agent-related reasoning. The paper concludes that text-only reasoning enhancement can provide a foundation for multimodal generalization.
Problem
3B LMMs face difficult visual-logical integration, ambiguous multimodal answers, scarce complex reasoning examples, and weakened foundational reasoning after multimodal training.
Method
LMM-R1 uses rule-based RL in two stages: text-only Foundational Reasoning Enhancement followed by multimodal generalization training.
Results
LMM-R1 improves average performance by 4.83% on five multimodal benchmarks and 4.5% on text-only benchmarks, with a 3.63% gain on Football Game tasks.
Takeaways & Limitations
Text-only reasoning enhancement establishes a foundation that generalizes to multimodal and agent-related reasoning domains.
Takeaways & Limitations
Agent-related tasks have data distributions significantly different from the first-stage RL training and were not encountered during pretraining.
Abstract
from arXiv · showhide
Enhancing reasoning in Large Multimodal Models (LMMs) faces unique challenges from the complex interplay between visual perception and logical reasoning, particularly in compact 3B-parameter architectures where architectural constraints limit reasoning capacity and modality alignment. While rule-based reinforcement learning (RL) excels in text-only domains, its multimodal extension confronts two critical barriers: (1) data limitations due to ambiguous answers and scarce complex reasoning examples, and (2) degraded foundational reasoning induced by multimodal pretraining. To address these challenges, we propose \textbf{LMM-R1}, a two-stage framework adapting rule-based RL for multimodal reasoning through \textbf{Foundational Reasoning Enhancement (FRE)} followed by \textbf{Multimodal Generalization Training (MGT)}. The FRE stage first strengthens reasoning abilities using text-only data with rule-based RL, then the MGT stage generalizes these reasoning capabilities to multimodal domains. Experiments on Qwen2.5-VL-Instruct-3B demonstrate that LMM-R1 achieves 4.83\% and 4.5\% average improvements over baselines in multimodal and text-only benchmarks, respectively, with a 3.63\% gain in complex Football Game tasks. These results validate that text-based reasoning enhancement enables effective multimodal generalization, offering a data-efficient paradigm that bypasses costly high-quality multimodal training data.
1. Introduction
LMM-R1 addresses multimodal reasoning challenges in compact 3B LMMs with a two-stage rule-based RL framework that first strengthens text-only reasoning, then generalizes it multimodally. Experiments report gains across text-only, multimodal, and agent-related benchmarks.
- Illustration: LMM-R1 correctly applies the Pythagorean theorem to determine a cone’s slant height, unlike the baseline LMM.Figure 1 presents this geometric problem as an example of improved mathematical reasoning.
- Motivation: 3B LMMs face intertwined visual-perception and logical-reasoning challenges, while limited capacity and scarce high-quality multimodal reasoning data constrain training.Multimodal tasks also involve ambiguous answers and relatively few complex reasoning examples.
- Approach: LMM-R1 combines Foundational Reasoning Enhancement with Multimodal Generalization Training in a two-stage rule-based RL framework.FRE uses abundant high-quality text-only data; MGT continues training on limited complex multimodal reasoning tasks.
- Evaluation: Sokoban and football tasks evaluate sequential decision-making, planning, and multi-image processing in agent-related visual environments.The agent-related domain includes goal identification, path planning, and multi-image processing.
- Results: 4.5%/4.83%/3.63% performance increases over the baseline are reported on text-only/five multimodal/Football Game benchmarks, respectively.The experiments use Qwen2.5-VL-Instruct-3B as the baseline model.
- Findings: Text-only RL improves multimodal reasoning across general and agent-related domains, whereas direct rule-based RL on multimodal data yields limited reasoning improvement.The authors attribute this pattern to the difficulty of generating high-quality long reasoning processes after direct multimodal training.
2. Related Work
Related work covers the development of large multimodal models and approaches for improving reasoning in language and multimodal systems.
- Large Multimodal Models: Large multimodal models integrate modalities such as vision with language models for visual question answering, instruction following, grounding, and visual understanding.The literature includes encoder-LLM alignment, visual instruction tuning, grounding, scaling, and unified vision understanding and generation.
3. Preliminaries
The framework trains LMMs with PPO and a rule-based reward combining response format compliance with answer accuracy.
- Reinforcement Learning: PPO maximizes an objective balancing expected reward against KL divergence from a fixed initial policy.The policy generates answers conditioned on text prompts and optional image inputs.
- Reward Function: Format reward checks whether reasoning and final answers use the required <think></think> and <answer></answer> structure.This structure supports explicit reasoning and readable evaluation.
- Reward Function: Accuracy reward uses symbolic verification to compare parsed model answers and ground-truth solutions for equivalence rather than exact string matching.Equivalent expressions can therefore receive matching correctness treatment.
- Reward Function: The reward function is r(y) = α·rf(y)+ra(y), combining format and accuracy rewards.α controls the relative importance of format versus accuracy.
4. LMM-R1: Two-Stage Rule-based RL
LMM-R1 first enhances foundational reasoning with text-only rule-based RL, then generalizes it through continued training on multimodal and agent-related tasks.
- Foundational Reasoning Enhancement: Stage 1 applies rule-based RL to high-quality text-only reasoning data to increase foundational reasoning ability.The text-only problems are described as diverse, verifiable, and demanding complex reasoning.
- Multimodal Generalization Training: Stage 2 continues RL across general multimodal and agent-related reasoning domains to improve multimodal reasoning.The two domains test different aspects of multimodal intelligence.
- General Multimodal Domain: Geometric multimodal training bridges mathematical reasoning from FRE with visual-perception challenges.The geometric domain requires extracting visual information and applying mathematical reasoning.
- Agent-Related Reasoning Domain: Sokoban evaluates spatial reasoning and planning through box-pushing tasks, while football evaluates strategic decisions in a multi-agent environment.Football additionally tests cooperation and interaction based on teammate and opponent positions.
- Evaluation: Table 1 reports results across high-level text-only reasoning, multimodal reasoning, and general multimodal benchmarks, including an MM Avg column.The table identifies best and second-best results through bolding and underlining.
- Agent-Related Reasoning Domain: The agent-related tasks differ substantially from the first-stage training distribution and were not encountered during pretraining.Continued Sokoban training with rule-based RL is used to enhance agent capabilities.
5. Experiments
Experiments compare text-only and multimodal rule-based RL across general, geometry, and agent benchmarks. The results favor FRE-Text followed by multimodal generalization for preserving reasoning while improving visual and agent capabilities.
- Text-Only Benchmarks: FRE-Text improves text-only performance by 4.29% overall, including 2.0% on MATH500 and 6.57% on GPQA.
- Multimodal Benchmarks: FRE-Multi improves general multimodal tasks more than FRE-Text, with gains of 3.5% versus 0.7% on MathVista and 7.36% versus 3.75% on MM-Star.
- Multimodal Benchmarks: FRE-Text improves reasoning-dominated multimodal benchmarks by 5.34% on OlympiadBench, 2.17% on MathVision, and 4.19% on MathVerse.
- Geometry Domain: MGT-Geo surpasses the baseline by 3.21% on multimodal benchmarks and improves over FRE-Text by 3.35% on MathVision and 2.97% on MathVerse.
- Perception-Reasoning Domain: MGT-PerceReason increases multimodal benchmark performance over FRE-Text by 1.6%, including gains of 1.8% on MathVista and 2.88% on MM-Star without sacrificing reasoning performance.
- Agent Domain Generalization: Text-only reasoning transfers stably to agent environments, while additional Sokoban rule-based RL training further improves agent performance in a 3B-parameter model.
6. Conclusion
LMM-R1 enhances multimodal reasoning in 3B LMMs through two-stage rule-based RL. Text-only reasoning provides a foundation for multimodal generalization, while the framework also improves complex planning in agent-related domains.
- 6. Conclusion: LMM-R1 uses Foundational Reasoning Enhancement followed by Multimodal Generalization Training to improve multimodal reasoning.The framework applies rule-based RL across these two stages.
- 6. Conclusion: Text-only reasoning establishes a foundation for multimodal generalization and outperforms training solely on text or multimodal data.The passage describes this interaction as a synergistic effect.
- 6. Conclusion: LMM-R1 notably improves performance in complex planning tasks within agent-related domains.The conclusion identifies agent-related domains as an additional area of improvement.
- 6. Conclusion: Future work extends the framework to additional LMMs and seeks methods for synthesizing high-quality multimodal reasoning data.These directions address broader model coverage and multimodal reasoning-data creation.
A.1. Datasets
The appendix describes verifiable datasets for text-only, multimodal, geometry, and Sokoban reasoning. These datasets combine filtered existing problems with generated environments for agent training.
- A.1. Datasets: DeepScaleR-Preview provides 40.3k filtered, high-quality text-only math reasoning problems with verifiable answers.Its sources include AIME, AMC, Omni-MATH, and STILL, with redundant problems removed.
- A.1. Datasets: VerMulti-65K contains 65k verifiable problems filtered from MathV360K for multimodal reasoning training.The problems come from various sources listed in the appendix table.
- A.1. Datasets: Table A4 reports statistics for VerMulti.The supplied appendix text provides the table’s title but no individual statistics.
- A.1. Datasets: VerMulti-Geo15K contains 15k geometry problems from MathV360K for multimodal geometry training.Its sources include GEOS, Geometry3K, GeoQA+, UniGeo, and TQA.
- A.1. Datasets: The Sokoban training data comprises 11,500 generated environments spanning multiple gym-sokoban difficulty settings.The environments include 5,000 each from sokoban-small-v0 and sokoban-small-v1, plus 500 each from three additional settings.
A.2. Training Hyper-Parameters
Training uses OpenRLHF with a shared RL hyper-parameter setup, while supervised text training runs for two epochs. MM-Star results assess whether multimodal generalization affects perception.
- A.2. Training Hyper-Parameters: The RL training infrastructure is built on OpenRLHF and uses the hyper-parameters listed in Table A7.The passage does not enumerate the individual RL hyper-parameter values.
- A.2. Training Hyper-Parameters: SFT-Text is trained for 2 epochs with batch-size=256, learning-rate=5e-6, and warmup-ratio=0.03.These settings specify the supervised text-training schedule and optimization parameters.
- A.2. Training Hyper-Parameters: MM-Star split results evaluate performance across coarse and fine-grained perception categories.Coarse perception concerns basic object recognition, while fine-grained perception concerns detailed visual discrimination.
- A.2. Training Hyper-Parameters: MGT improves the perception ability of FRE-Text on the MM-Star evaluation.The passage presents this as validation on general visual tasks.
C.1. Text-Only Benchmarks
The evaluation uses standardized toolkits and task-specific protocols across text-only, multimodal, and agent-related benchmarks. It specifies extraction rules, generation limits, prompting choices, and planner designs for reproducible comparisons.
- C.1. Text-Only Benchmarks: LightEval evaluates text-only mathematical reasoning on MATH-500 and GPQA-Diamond.Custom tasks use prompts encouraging step-by-step reasoning and clear answer formatting.
- C.1. Text-Only Benchmarks: Text-only evaluation permits up to 32,768 generated tokens to accommodate extensive reasoning chains.This is the stated maximum generation length for the evaluation.
- C.1. Text-Only Benchmarks: MATH-500 answers are extracted from LaTeX expressions, while GPQA-Diamond answers are extracted as options A, B, C, or D.Both metrics use LightEval’s multilingual extractive matching framework.
- C.2. Multimodal Benchmark: LMMs-Eval covers multimodal reasoning benchmarks including MathVision, MathVerse, MathVista, OlympiadBench, and MM-Star.The toolkit is described as comprehensive and lightweight.
- C.2. Multimodal Benchmark: Multimodal evaluation uses vLLM for deployment and Qwen2.5-14B-Instruct as a judge for answer extraction.The judge follows a prompt specified in Appendix E.
- C.2. Multimodal Benchmark: Baseline models do not use the proposed system prompt because applying the same prompt degrades their performance.This is the stated fairness choice in the evaluation setup.
- Agent-Related Benchmarks: Sokoban and football evaluations follow MageBench and use global-planner and online-planner agent designs.Sokoban-Global tests mental simulation, whereas Football-Online tests real-time decisions under changing game states.
D. Training Curves
The appendix presents training-curve figures for FRE-Text, FRE-Multi, MGT-Geo, and MGT-PerceReason, alongside the prompts and answer-extraction formats used for training and evaluation.
- Training curves are provided for FRE-Text, FRE-Multi, MGT-Geo, and MGT-PerceReason.
- The system prompts instruct models to reason step by step and place answers in designated formats.
- Text-only math prompts require a boxed final answer on the last line, while multiple-choice prompts require an answer letter.
- Answer extraction covers integers, floating-point values, Python lists, and multiple-choice option letters.
F. Case Study
The case studies compare Qwen2.5-VL outputs with FRE-enhanced outputs across text-only and multimodal reasoning examples, including algebra, logarithms, charts, and image understanding.
- Qwen2.5-VL CoT and FRE-Text: The quadratic-equation case derives the transformed equation -11x^2 + 39x + 5 = 0 and reports the solution sum as 39/11.
- Qwen2.5-VL CoT and FRE-Text: The logarithm case reports one valid base under Qwen2.5-VL reasoning but four valid bases after FRE-Text.
- Qwen2.5-VL CoT and FRE-Text: The FRE-Text solution enumerates b values 3, 9, 27, and 729 by considering factor pairs of 6.
- Qwen2.5-VL and FRE-Multi: For the stem-and-leaf plot, Qwen2.5-VL reports a median of 56, whereas FRE-Multi reports 60.
- Qwen2.5-VL v.s. FRE-Multi: Both Qwen2.5-VL and FRE-Multi report the chart value for ‘legs’ under ‘grade’ as 4.
- Qwen2.5-VL and FRE-Multi: The vehicle-count case shows differing reasoning: Qwen2.5-VL gives 4, while FRE-Multi identifies five wheeled vehicles.