PaperlayerView exact revision on arXiv

Source-linked AI summary

LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RL

Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, Xu Yang

arXiv:2503.07536v2cs.CLcs.AI

TL;DR

Compact 3B LMMs struggle with multimodal reasoning because visual tasks combine perception and logic, while answers can be ambiguous and complex reasoning data are scarce. LMM-R1 addresses this with text-only Foundational Reasoning Enhancement followed by Multimodal Generalization Training, achieving broad benchmark gains and improved agent-related reasoning. The paper concludes that text-only reasoning enhancement can provide a foundation for multimodal generalization.

  • Problem

    3B LMMs face difficult visual-logical integration, ambiguous multimodal answers, scarce complex reasoning examples, and weakened foundational reasoning after multimodal training.

    Source: §1 · p. 2

  • Method

    LMM-R1 uses rule-based RL in two stages: text-only Foundational Reasoning Enhancement followed by multimodal generalization training.

    Source: §1 · p. 2

  • Results

    LMM-R1 improves average performance by 4.83% on five multimodal benchmarks and 4.5% on text-only benchmarks, with a 3.63% gain on Football Game tasks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Text-only reasoning enhancement establishes a foundation that generalizes to multimodal and agent-related reasoning domains.

    Source: §6 · p. 9

    Agent-related tasks have data distributions significantly different from the first-stage RL training and were not encountered during pretraining.

    Source: §4.2.2 · p. 5

Abstract

from arXiv · show

Enhancing reasoning in Large Multimodal Models (LMMs) faces unique challenges from the complex interplay between visual perception and logical reasoning, particularly in compact 3B-parameter architectures where architectural constraints limit reasoning capacity and modality alignment. While rule-based reinforcement learning (RL) excels in text-only domains, its multimodal extension confronts two critical barriers: (1) data limitations due to ambiguous answers and scarce complex reasoning examples, and (2) degraded foundational reasoning induced by multimodal pretraining. To address these challenges, we propose \textbf{LMM-R1}, a two-stage framework adapting rule-based RL for multimodal reasoning through \textbf{Foundational Reasoning Enhancement (FRE)} followed by \textbf{Multimodal Generalization Training (MGT)}. The FRE stage first strengthens reasoning abilities using text-only data with rule-based RL, then the MGT stage generalizes these reasoning capabilities to multimodal domains. Experiments on Qwen2.5-VL-Instruct-3B demonstrate that LMM-R1 achieves 4.83\% and 4.5\% average improvements over baselines in multimodal and text-only benchmarks, respectively, with a 3.63\% gain in complex Football Game tasks. These results validate that text-based reasoning enhancement enables effective multimodal generalization, offering a data-efficient paradigm that bypasses costly high-quality multimodal training data.

1. Introduction

LMM-R1 addresses multimodal reasoning challenges in compact 3B LMMs with a two-stage rule-based RL framework that first strengthens text-only reasoning, then generalizes it multimodally. Experiments report gains across text-only, multimodal, and agent-related benchmarks.

Illustration

  • LMM-R1 correctly applies the Pythagorean theorem to determine a cone’s slant height, unlike the baseline LMM.Figure 1 presents this geometric problem as an example of improved mathematical reasoning.Source: Figure 1 · p. 1, §1 · p. 2

Motivation

  • 3B LMMs face intertwined visual-perception and logical-reasoning challenges, while limited capacity and scarce high-quality multimodal reasoning data constrain training.Multimodal tasks also involve ambiguous answers and relatively few complex reasoning examples.Source: §1 · p. 2

Approach

  • LMM-R1 combines Foundational Reasoning Enhancement with Multimodal Generalization Training in a two-stage rule-based RL framework.FRE uses abundant high-quality text-only data; MGT continues training on limited complex multimodal reasoning tasks.Source: §1 · p. 2, Figure 2 · p. 2

Evaluation

  • Sokoban and football tasks evaluate sequential decision-making, planning, and multi-image processing in agent-related visual environments.The agent-related domain includes goal identification, path planning, and multi-image processing.Source: §1 · p. 3

Results

  • 4.5%/4.83%/3.63% performance increases over the baseline are reported on text-only/five multimodal/Football Game benchmarks, respectively.The experiments use Qwen2.5-VL-Instruct-3B as the baseline model.Source: §1 · p. 3

Findings

  • Text-only RL improves multimodal reasoning across general and agent-related domains, whereas direct rule-based RL on multimodal data yields limited reasoning improvement.The authors attribute this pattern to the difficulty of generating high-quality long reasoning processes after direct multimodal training.Source: §1 · pp. 2–3

2. Related Work

Related work covers the development of large multimodal models and approaches for improving reasoning in language and multimodal systems.

Large Multimodal Models

  • Large multimodal models integrate modalities such as vision with language models for visual question answering, instruction following, grounding, and visual understanding.The literature includes encoder-LLM alignment, visual instruction tuning, grounding, scaling, and unified vision understanding and generation.Source: §2 · p. 3

3. Preliminaries

The framework trains LMMs with PPO and a rule-based reward combining response format compliance with answer accuracy.

Reinforcement Learning

  • PPO maximizes an objective balancing expected reward against KL divergence from a fixed initial policy.The policy generates answers conditioned on text prompts and optional image inputs.Source: §3.1 · p. 3

Reward Function

  • Format reward checks whether reasoning and final answers use the required <think></think> and <answer></answer> structure.This structure supports explicit reasoning and readable evaluation.Source: §3.2 · p. 3
  • Accuracy reward uses symbolic verification to compare parsed model answers and ground-truth solutions for equivalence rather than exact string matching.Equivalent expressions can therefore receive matching correctness treatment.Source: §3.2 · p. 4
  • The reward function is r(y) = α·rf(y)+ra(y), combining format and accuracy rewards.α controls the relative importance of format versus accuracy.Source: §3.2 · p. 4

4. LMM-R1: Two-Stage Rule-based RL

LMM-R1 first enhances foundational reasoning with text-only rule-based RL, then generalizes it through continued training on multimodal and agent-related tasks.

Foundational Reasoning Enhancement

  • Stage 1 applies rule-based RL to high-quality text-only reasoning data to increase foundational reasoning ability.The text-only problems are described as diverse, verifiable, and demanding complex reasoning.Source: §4 · p. 4, §4.1 · p. 4

Multimodal Generalization Training

  • Stage 2 continues RL across general multimodal and agent-related reasoning domains to improve multimodal reasoning.The two domains test different aspects of multimodal intelligence.Source: §4.2 · p. 4

General Multimodal Domain

  • Geometric multimodal training bridges mathematical reasoning from FRE with visual-perception challenges.The geometric domain requires extracting visual information and applying mathematical reasoning.Source: §4.2.1 · p. 4

Agent-Related Reasoning Domain

  • Sokoban evaluates spatial reasoning and planning through box-pushing tasks, while football evaluates strategic decisions in a multi-agent environment.Football additionally tests cooperation and interaction based on teammate and opponent positions.Source: §4.2.2 · pp. 4–5

Evaluation

  • Table 1 reports results across high-level text-only reasoning, multimodal reasoning, and general multimodal benchmarks, including an MM Avg column.The table identifies best and second-best results through bolding and underlining.Source: Table 1 · p. 5

Agent-Related Reasoning Domain

  • The agent-related tasks differ substantially from the first-stage training distribution and were not encountered during pretraining.Continued Sokoban training with rule-based RL is used to enhance agent capabilities.Source: §4.2.2 · p. 5

5. Experiments

Experiments compare text-only and multimodal rule-based RL across general, geometry, and agent benchmarks. The results favor FRE-Text followed by multimodal generalization for preserving reasoning while improving visual and agent capabilities.

Text-Only Benchmarks

  • FRE-Text improves text-only performance by 4.29% overall, including 2.0% on MATH500 and 6.57% on GPQA.Source: §5.2.1 · p. 6

Multimodal Benchmarks

  • FRE-Multi improves general multimodal tasks more than FRE-Text, with gains of 3.5% versus 0.7% on MathVista and 7.36% versus 3.75% on MM-Star.Source: §5.2.1 · p. 6
  • FRE-Text improves reasoning-dominated multimodal benchmarks by 5.34% on OlympiadBench, 2.17% on MathVision, and 4.19% on MathVerse.Source: §5.2.1 · p. 6

Geometry Domain

  • MGT-Geo surpasses the baseline by 3.21% on multimodal benchmarks and improves over FRE-Text by 3.35% on MathVision and 2.97% on MathVerse.Source: §5.2.1 · p. 6

Perception-Reasoning Domain

  • MGT-PerceReason increases multimodal benchmark performance over FRE-Text by 1.6%, including gains of 1.8% on MathVista and 2.88% on MM-Star without sacrificing reasoning performance.Source: §5.2.1 · p. 6

Agent Domain Generalization

  • Text-only reasoning transfers stably to agent environments, while additional Sokoban rule-based RL training further improves agent performance in a 3B-parameter model.Source: §5 · p. 5, §5.2.2 · p. 7

6. Conclusion

LMM-R1 enhances multimodal reasoning in 3B LMMs through two-stage rule-based RL. Text-only reasoning provides a foundation for multimodal generalization, while the framework also improves complex planning in agent-related domains.

6. Conclusion

  • LMM-R1 uses Foundational Reasoning Enhancement followed by Multimodal Generalization Training to improve multimodal reasoning.The framework applies rule-based RL across these two stages.Source: §6 · p. 9
  • Text-only reasoning establishes a foundation for multimodal generalization and outperforms training solely on text or multimodal data.The passage describes this interaction as a synergistic effect.Source: §6 · p. 9
  • LMM-R1 notably improves performance in complex planning tasks within agent-related domains.The conclusion identifies agent-related domains as an additional area of improvement.Source: §6 · p. 9
  • Future work extends the framework to additional LMMs and seeks methods for synthesizing high-quality multimodal reasoning data.These directions address broader model coverage and multimodal reasoning-data creation.Source: §6 · p. 9

A.1. Datasets

The appendix describes verifiable datasets for text-only, multimodal, geometry, and Sokoban reasoning. These datasets combine filtered existing problems with generated environments for agent training.

A.1. Datasets

  • DeepScaleR-Preview provides 40.3k filtered, high-quality text-only math reasoning problems with verifiable answers.Its sources include AIME, AMC, Omni-MATH, and STILL, with redundant problems removed.Source: §A.1 · p. 13
  • VerMulti-65K contains 65k verifiable problems filtered from MathV360K for multimodal reasoning training.The problems come from various sources listed in the appendix table.Source: §A.1 · p. 13
  • Table A4 reports statistics for VerMulti.The supplied appendix text provides the table’s title but no individual statistics.Source: Table A4 · p. 13
  • VerMulti-Geo15K contains 15k geometry problems from MathV360K for multimodal geometry training.Its sources include GEOS, Geometry3K, GeoQA+, UniGeo, and TQA.Source: §A.1 · p. 13
  • The Sokoban training data comprises 11,500 generated environments spanning multiple gym-sokoban difficulty settings.The environments include 5,000 each from sokoban-small-v0 and sokoban-small-v1, plus 500 each from three additional settings.Source: §A.1 · p. 13

A.2. Training Hyper-Parameters

Training uses OpenRLHF with a shared RL hyper-parameter setup, while supervised text training runs for two epochs. MM-Star results assess whether multimodal generalization affects perception.

A.2. Training Hyper-Parameters

  • The RL training infrastructure is built on OpenRLHF and uses the hyper-parameters listed in Table A7.The passage does not enumerate the individual RL hyper-parameter values.Source: §A.2 · p. 13
  • SFT-Text is trained for 2 epochs with batch-size=256, learning-rate=5e-6, and warmup-ratio=0.03.These settings specify the supervised text-training schedule and optimization parameters.Source: §A.2 · p. 13
  • MM-Star split results evaluate performance across coarse and fine-grained perception categories.Coarse perception concerns basic object recognition, while fine-grained perception concerns detailed visual discrimination.Source: B. Detailed Performance on MM-Star · p. 13, Table A6 · p. 13
  • MGT improves the perception ability of FRE-Text on the MM-Star evaluation.The passage presents this as validation on general visual tasks.Source: B. Detailed Performance on MM-Star · p. 13

C.1. Text-Only Benchmarks

The evaluation uses standardized toolkits and task-specific protocols across text-only, multimodal, and agent-related benchmarks. It specifies extraction rules, generation limits, prompting choices, and planner designs for reproducible comparisons.

C.1. Text-Only Benchmarks

  • LightEval evaluates text-only mathematical reasoning on MATH-500 and GPQA-Diamond.Custom tasks use prompts encouraging step-by-step reasoning and clear answer formatting.Source: §C.1 · p. 13
  • Text-only evaluation permits up to 32,768 generated tokens to accommodate extensive reasoning chains.This is the stated maximum generation length for the evaluation.Source: §C.1 · p. 14
  • MATH-500 answers are extracted from LaTeX expressions, while GPQA-Diamond answers are extracted as options A, B, C, or D.Both metrics use LightEval’s multilingual extractive matching framework.Source: §C.1 · p. 14

C.2. Multimodal Benchmark

  • LMMs-Eval covers multimodal reasoning benchmarks including MathVision, MathVerse, MathVista, OlympiadBench, and MM-Star.The toolkit is described as comprehensive and lightweight.Source: §C.2 · p. 14
  • Multimodal evaluation uses vLLM for deployment and Qwen2.5-14B-Instruct as a judge for answer extraction.The judge follows a prompt specified in Appendix E.Source: §C.2 · p. 14
  • Baseline models do not use the proposed system prompt because applying the same prompt degrades their performance.This is the stated fairness choice in the evaluation setup.Source: §C.2 · p. 14

Agent-Related Benchmarks

  • Sokoban and football evaluations follow MageBench and use global-planner and online-planner agent designs.Sokoban-Global tests mental simulation, whereas Football-Online tests real-time decisions under changing game states.Source: §C.3 · p. 14

D. Training Curves

The appendix presents training-curve figures for FRE-Text, FRE-Multi, MGT-Geo, and MGT-PerceReason, alongside the prompts and answer-extraction formats used for training and evaluation.

F. Case Study

The case studies compare Qwen2.5-VL outputs with FRE-enhanced outputs across text-only and multimodal reasoning examples, including algebra, logarithms, charts, and image understanding.

Qwen2.5-VL CoT and FRE-Text

  • The quadratic-equation case derives the transformed equation -11x^2 + 39x + 5 = 0 and reports the solution sum as 39/11.Source: F. Case Study · pp. 19–20
  • The logarithm case reports one valid base under Qwen2.5-VL reasoning but four valid bases after FRE-Text.Source: F. Case Study · pp. 20–21
  • The FRE-Text solution enumerates b values 3, 9, 27, and 729 by considering factor pairs of 6.Source: F. Case Study · pp. 20–21

Qwen2.5-VL and FRE-Multi

Qwen2.5-VL v.s. FRE-Multi

  • Both Qwen2.5-VL and FRE-Multi report the chart value for ‘legs’ under ‘grade’ as 4.Source: F. Case Study · p. 22

Qwen2.5-VL and FRE-Multi

  • The vehicle-count case shows differing reasoning: Qwen2.5-VL gives 4, while FRE-Multi identifies five wheeled vehicles.Source: F. Case Study · p. 23

Related papers

Loading 2503.07536v2…