PaperlayerView exact revision on arXiv

Source-linked AI summary

Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models

Yunxin Li, Zhenyu Liu, Zitao Li, Xuanyu Zhang, Zhenran Xu, Xinyu Chen, Haoyuan Shi, Shenyuan Jiang, Xintong Wang, Jifang Wang, Shouzheng Huang, Xinping Zhao, Borui Jiang, Lanqing Hong, Longyue Wang, Zhuotao Tian, Baoxing Huai, Wenhan Luo, Weihua Luo, Zheng Zhang, Baotian Hu, Min Zhang

arXiv:2505.04921v2cs.CVcs.CL

TL;DR

Multimodal reasoning research seeks robust intelligence across open, uncertain environments, but current models remain limited in generalization, reasoning depth, and agentic behavior. The survey organizes this field into a developmental roadmap from modular reasoning through multimodal chain-of-thought to native multimodal reasoning models, while identifying progress and persistent constraints. It points toward scalable, agentic, and adaptive reasoning and planning across complex real-world environments.

  • Problem

    Multimodal reasoning remains limited in generalization, reasoning depth, agent-like behavior, and effective integration of cross-modal interactions.

  • Method

    The paper surveys about 700 publications and proposes a three-stage roadmap from modular reasoning to multimodal chain-of-thought and long-horizon System-2 reasoning, alongside an analysis of native multimodal reasoning models.

  • Results

    LMRMs generally lead current agent-benchmark performance, but state-of-the-art models still fall short of human-level reliability on complex, open-ended tasks.

  • Takeaways & Limitations

    Native large multimodal reasoning models should support multimodal agentic reasoning, omni-modal understanding and generative reasoning, and coherent cross-modal outputs.

  • Takeaways & Limitations

    Current language-centric models are constrained by predominant focus on vision and language, limiting applicability to settings involving audio, tactile signals, sensor streams, and temporal sequences.

Abstract

from arXiv · show

Reasoning lies at the heart of intelligence, shaping the ability to make decisions, draw conclusions, and generalize across domains. In artificial intelligence, as systems increasingly operate in open, uncertain, and multimodal environments, reasoning becomes essential for enabling robust and adaptive behavior. Large Multimodal Reasoning Models (LMRMs) have emerged as a promising paradigm, integrating modalities such as text, images, audio, and video to support complex reasoning capabilities and aiming to achieve comprehensive perception, precise understanding, and deep reasoning. As research advances, multimodal reasoning has rapidly evolved from modular, perception-driven pipelines to unified, language-centric frameworks that offer more coherent cross-modal understanding. While instruction tuning and reinforcement learning have improved model reasoning, significant challenges remain in omni-modal generalization, reasoning depth, and agentic behavior. To address these issues, we present a comprehensive and structured survey of multimodal reasoning research, organized around a four-stage developmental roadmap that reflects the field's shifting design philosophies and emerging capabilities. First, we review early efforts based on task-specific modules, where reasoning was implicitly embedded across stages of representation, alignment, and fusion. Next, we examine recent approaches that unify reasoning into multimodal LLMs, with advances such as Multimodal Chain-of-Thought (MCoT) and multimodal reinforcement learning enabling richer and more structured reasoning chains. Finally, drawing on empirical insights from challenging benchmarks and experimental cases of OpenAI O3 and O4-mini, we discuss the conceptual direction of native large multimodal reasoning models (N-LMRMs), which aim to support scalable, agentic, and adaptive reasoning and planning in complex, real-world environments.

1. Introduction

Reasoning is essential for robust intelligence in dynamic, uncertain, multimodal environments, yet current multimodal systems still face limits in generalization, reasoning depth, and agent-like behavior. This survey addresses the gap with a structured roadmap spanning modular systems, language-centric reasoning, and future directions.

  • Reasoning enables adaptive responses, logical inference, knowledge generalization, and navigation of complex challenges across environments.
  • Multimodal reasoning evolved from perception-driven modular pipelines toward language models that unify multimodal understanding and reasoning.
  • Instruction tuning and reinforcement learning have improved multimodal reasoning performance, but generalization, reasoning depth, and agent-like behavior remain limited.
  • Previous surveys undercovered reinforcement-enhanced multimodal reasoning and the technical prospects of LMRMs.
  • The survey proposes a structured roadmap covering modular reasoning, language-centric short reasoning, and language-centric long reasoning.

2. Evolving Paradigms of Multimodal Reasoning and Discussion

Multimodal reasoning has progressed from modular systems with implicit reasoning to language-centric and increasingly native multimodal paradigms. The survey organizes this evolution into stages and introduces N-LMRMs as a forward-looking framework for omnimodal, goal-driven reasoning.

  • N-LMRMs are proposed as models in which reasoning emerges natively from omnimodal perception, interaction, and goal-driven cognition rather than being retrofitted onto language models.
  • The survey presents a three-stage roadmap from modular reasoning to multimodal chain-of-thought and long-horizon System-2 reasoning.
  • Stage 1: Early systems decomposed reasoning into separate representation, alignment, and fusion components using supervised modular architectures.
  • Stage 1: Early multimodal unification emphasized visual representation and cross-modal fusion, often at the expense of deeper language semantics and generalized reasoning.
  • Stage 2: Multimodal large language models shifted the field toward end-to-end, language-centric frameworks with strong performance on visual reasoning tasks.

2. Evolving Paradigms of Multimodal Reasoning and Discussion

Multimodal chain-of-thought made reasoning more explicit, but its short and reactive character remains inadequate for complex real-world tasks. Later work emphasizes cross-modal chains, longer structured procedures, and reinforcement learning to support more deliberative reasoning.

  • Early MLLMs relied on surface pattern matching and static retrieval, motivating chain-of-thought methods that expose intermediate reasoning steps.
  • MCoT extends chain-of-thought to multimodal inputs through prompting, structured decomposition, tools, and retrieval augmentation.
  • MCoT reasoning remains predominantly short and reactive, limiting abstraction, compositionality, and planning.
  • System-2-inspired approaches target slower, deliberate, and methodologically structured reasoning for long-horizon multimodal tasks.
  • Cross-modal reasoning chains combine visual, auditory, and linguistic signals as joint substrates for inference and richer semantic grounding.
  • Longer guided chains let models decompose complex tasks and apply transferable procedures across contexts, while reinforcement learning improves planning, robustness, and adaptive generalization.
  • Together, these developments mark a transition from reactive to deliberative reasoning paradigms.
  • Language-centric LMRMs remain constrained by limited modality coverage and deficiencies in interactive, long-horizon multimodal control.

3. Roadmap of Multimodal Reasoning Models

The roadmap points from current models that can generate extended reasoning chains toward native multimodal systems designed for adaptive interaction and planning. N-LMRMs aim to unify multimodal understanding, generation, and agentic reasoning end to end.

  • Current models can produce extended reasoning chains in static settings but remain underdeveloped in real-time, iterative interaction with dynamic environments.
  • N-LMRMs are envisioned as an end-to-end architecture unifying multimodal understanding, generation, and agentic reasoning.
  • The proposed evolution uses unified representations and large-scale synthetic data to support reasoning and planning across modality interactions.
  • The trajectory runs from modular perception-driven systems toward more unifying, adaptive, and comprehensive high-level AI systems.

3.1 Stage 1 Perception Driven Modular Reasoning - Developing Task-Specific Reasoning Modules

Stage 1 developed multimodal reasoning through task-specific modular pipelines that separated representation, alignment, fusion, and reasoning. Later pretrained vision-language models unified multimodal processing but often retained implicit, classification-oriented reasoning.

  • Stage 1 Perception-Driven Modular Reasoning: Early systems used CNNs, LSTMs, and supervised modular components to decompose multimodal reasoning into representation, alignment, fusion, and inference stages.
  • Stage 1 Perception-Driven Modular Reasoning: Neural Module Networks dynamically assembled task-specific modules, while hierarchical co-attention used modular cross-modal attention to align question semantics with image features.
  • Vision-Language Models-based Modular Reasoning: Pretrained vision-language models introduced Transformer-based multimodal representation, perception, fusion, and inference using large-scale image-text data.
  • Vision-Language Models-based Modular Reasoning: Dual-encoder, single-Transformer, and vision-encoder-LLM paradigms respectively aligned separate streams, jointly encoded visual-text inputs, or projected visual features into an LLM.

3.2 Stage 2 Language-Centric Short Reasoning - System-1 Reasoning

Stage 2 extended multimodal reasoning beyond implicit inference through short, language-centric chains. Prompting, supervised structural procedures, and external tools provided increasingly explicit and organized reasoning workflows.

  • Stage 2 Language-Centric Short Reasoning: MCoT adds intermediate reasoning steps to improve cross-modal alignment, knowledge integration, and contextual grounding without extensive additional supervision.
  • Prompt-based MCoT: Prompt-based MCoT supports step-by-step multimodal reasoning with minimal additional training and includes visual grounding, rationale verification, and progressive video interpretation.
  • Structural Reasoning: Structural reasoning learns predefined reasoning patterns through supervised training, converting loosely guided inference into standardized, stage-wise processes.
  • Structural Reasoning: Rationale construction, defined procedures, and modality-specific structures organize reasoning through generated explanations, explicit perception-decision stages, and region-based grounding.
  • Externally Augmented Reasoning: Externally augmented reasoning adds search algorithms, textual tools, or expert modules to support flexible, scalable, and task-specialized workflows.

3.3 Stage 3 Language-Centric Long Reasoning - System-2 Thinking and Planning

Stage 3 develops longer, more deliberate multimodal reasoning by combining extended chains, cross-modal information, search or tool support, and reinforcement learning. These approaches target planning, reflection, and adaptive behavior beyond short System-1 reasoning.

  • Language-Centric Long Reasoning: System-2-style multimodal reasoning addresses the shallow depth and limited adaptability of structural methods through deliberate, compositional reasoning and explicit planning.
  • Cross-Modal Reasoning: Cross-modal reasoning integrates text, images, video, audio, and programmatic information through external tools, algorithms, and model-intrinsic capabilities.
  • Multimodal-O1: Multimodal-O1 approaches deepen Chain-of-Thought workflows with multi-stage generation, long-horizon reasoning, rationale-rich supervision, and search methods such as Beam Search or MCTS.
  • Multimodal-R1: Reinforcement-learning approaches use DPO, GRPO, preference data, multimodal feedback, and long-horizon objectives to optimize complex reasoning paths.

4 Towards Native Multimodal Reasoning Model

Current LMRMs show progress on complex reasoning and agent benchmarks but remain constrained in multimodal breadth, real-world interaction, and reliability. The paper therefore proposes N-LMRMs to unify multimodal understanding, generation, and agentic reasoning.

  • Motivation: Current LMRMs rely mainly on vision and language, limiting reasoning over diverse interleaved data and real-time interaction with dynamic environments.These constraints motivate broader multimodal integration and more advanced interactive reasoning.
  • Experimental Findings: Evaluation should extend beyond vision-language questions to omni-modal inputs and long-horizon interaction with external environments.The survey collects omni-modal and agentic benchmarks to analyze these capabilities.
  • Experimental Findings: Agent benchmarks show progress, but leading models still fall short of human-level reliability in grounding, long-horizon planning, tool integration, and cross-domain robustness.On BrowseComp, GPT-4o reaches 0.6% accuracy, GPT-4o with browsing tools reaches 1.9%, o1 reaches 9.9%, and OpenAI Deep Research reaches 51.5%.
  • Experimental Findings: OpenAI o3 handles complex multimodal problem-solving but exhibits errors in visual counting, file processing, multimedia generation, and the faithfulness of its reasoning.The case studies include an eight-minute-and-thirteen-second chain of thought and failures involving six-finger recognition, resume parsing, and red-panda generation.
  • Technical Prospects: N-LMRMs are proposed as end-to-end systems combining multimodal agentic reasoning with omni-modal understanding and generative reasoning.The proposed direction targets unified representations, heterogeneous data fusion, contextual interaction, hierarchical decomposition, strategic adaptation, and embodied learning.
  • Technical Prospects: The survey frames N-LMRMs as an initial step from capable reasoners toward autonomous agents, while identifying architectural heterogeneity, tool use, and continuous real-world learning as challenges.The proposed scope spans language, vision, audio, tactile signals, sensor readings, temporal sequences, and structured data.

5. Dataset and Benchmark

The survey organizes multimodal datasets and benchmarks by capability, covering understanding, generation, reasoning, and planning. Understanding benchmarks span visual and audio inputs, with visual evaluations ranging from basic perception to specialized and comprehensive tasks.

  • Datasets and benchmarks are organized into four capability categories: Understanding, Generation, Reasoning, and Planning.
  • Understanding: Multimodal understanding evaluates processing and interpretation of visual and auditory information for comprehension, reasoning, and generation.
  • Visual-Centric Understanding: Visual-centric benchmarks cover general understanding, document and chart interpretation, multilingual reasoning, video understanding, mathematical and scientific reasoning, and comprehensive evaluation.
  • Visual-Centric Understanding: General visual datasets have progressed from object recognition, attribute identification, and simple spatial reasoning toward more complex visual-language tasks.
  • Visual-Centric Understanding: Specialized benchmarks address spatial, cross-source, causal, contextual, educational, temporal, multilingual, and real-world visual understanding.
  • Audio-Centric Understanding: Audio-centric evaluations assess speech, environmental sounds, and music understanding, including accuracy, translation, emotion recognition, and general comprehension.

5.2 Multimodal Generation

The passage highlights evaluations of multimodal generation and their relevance to models’ versatility and real-world effectiveness, particularly for complex audio data.

  • Evaluations help assess models’ versatility and effectiveness in handling complex audio data.

5.2 Multimodal Generation

Multimodal generation is categorized by cross-modal transformation and joint multimodal creation. Its datasets and benchmarks target alignment, instruction execution, coherence, and increasingly complex multimodal interactions.

  • Cross-modal Generation: Cross-modal generation produces one modality from another, including text-to-image, text-to-video, and text-to-speech tasks.
  • Cross-modal Generation: Text-to-image datasets support general generation, instruction-based editing, multi-turn editing, and conditional generation from scene structures or multiple inputs.
  • Cross-modal Generation: Text-to-video datasets and benchmarks cover diverse content while evaluating relevance, coherence, and related generation qualities.
  • Cross-modal Generation: Text-to-speech datasets and benchmarks support audio-language-model development through speech-dialogue evaluation using word error rates and accuracy.
  • Cross-modal Generation: Cross-modal generation requires transforming and aligning information across modalities, with continuing emphasis on complex prompts, multi-step reasoning, and semantic alignment.
  • Joint Multimodal Generation: Joint multimodal generation simultaneously creates multiple modalities, requiring coherence and alignment across outputs such as text, images, audio, and video.
  • Joint Multimodal Generation: Interleaved image-text datasets and benchmarks train and assess contextually relevant, visually coherent multimodal content.
  • Joint Multimodal Generation: Text-to-multimodal systems combine cross-modal and joint data to produce multimodal stories and incorporate embodied world models.

5.3 Multimodal Reasoning

Multimodal reasoning integrates information across modalities to support inference, problem solving, and complex question answering. Benchmarks span general visual reasoning and specialized domains, with datasets covering commonsense, video, mathematical, embodied, physical, and technical capabilities.

  • Multimodal reasoning integrates information from multiple modalities to make inferences, solve problems, and answer complex questions.
  • Reasoning benchmarks divide into general visual reasoning and domain-specific reasoning, including technical abilities such as mathematical problem solving from visual input.
  • General Visual Reasoning: General visual reasoning requires perception, comprehension, analysis, logical deduction, extensive knowledge, and common sense across scenarios.
  • General Visual Reasoning: Visual reasoning benchmarks test commonsense, physical reasoning, video physical understanding, multilingual ability, multidisciplinary knowledge, and human-centric evaluation.
  • General Visual Reasoning: Training datasets support visual commonsense, action continuation, synthetic-scene reasoning, multimodal dialogue, dynamic video reasoning, and visual chain-of-thought tasks.
  • Domain-Specific Reasoning: Domain-specific benchmarks evaluate mathematical and scientific reasoning, chart interpretation, embodied navigation, interaction, spatial understanding, and manipulation.
  • Domain-Specific Reasoning: Physical-reasoning datasets assess physical laws and common sense, while world-model benchmarks evaluate simulation and reasoning about real-world dynamics and interactions.
  • Domain-Specific Reasoning: Specialized datasets provide explicit reasoning paths for vision-centric instruction tuning and reinforcement learning, knowledge-graph generation, and multimodal scientific reasoning.

5.4 Multimodal Planning

Multimodal planning benchmarks evaluate agents across visual, textual, and interactive environments, spanning general and domain-specific reasoning tasks. They cover web navigation, GUIs, embodied settings, and open-ended simulations, while remaining limited in long-horizon planning, noisy inputs, and real-world adaptability.

  • Task Categories: Domain-specific benchmarks support specialized multimodal reasoning applications in areas such as mathematics, robotics, and physical laws.The survey distinguishes these tasks from general visual reasoning, which applies logic and common sense to visual content.
  • Benchmark Scope: Multimodal planning benchmarks test agents’ integration of diverse inputs while performing complex, multi-step tasks.The evaluated inputs include visual, textual, and interactive data.
  • GUI Navigation: GUI benchmarks assess visual-language grounding and multi-step reasoning for digital-interface navigation and information extraction.WebArena and Mind2Web provide realistic web environments, while Mind2Web also tests cross-website generalizability.
  • Embodied Environments: Embodied and simulated benchmarks evaluate planning in dynamic environments requiring adaptation to physical or virtual worlds.MineDojo supports open-ended Minecraft tasks involving object interaction, navigation, and resource management.

5.5 Evaluation Method

Multimodal reasoning evaluation combines answer matching, option selection, model-based scoring, and agentic assessment. These methods trade controlled comparison against the flexibility needed for open-ended and multimodal tasks, while broader planning evaluation still faces adaptability challenges.

  • Evaluation Challenges: Planning evaluation still struggles with long-horizon tasks, noisy inputs, and adaptation to real-world conditions.Future benchmarks are encouraged to include open-world environments, human feedback, and collaborative planning.
  • Evaluation Methods: The main evaluation methods are Exact/Fuzzy Match, Option Matching, LLM/MLLM Scoring, and Agentic Evaluation.Together, they span rule-based answer comparison, constrained selection, model-based judging, and tool-assisted assessment.
  • Exact/Fuzzy Matching: Exact/Fuzzy Matching compares rule-processed predictions with multiple human-annotated candidate answers in open-ended VQA datasets.VQAv2 and OKVQA are cited as examples of datasets using this approach.
  • Options Matching: Option Matching improves fairness and accuracy by requiring models to select among candidate options when answer diversity limits exact matching.The method also addresses potential preference for a specific option during selection.
  • LLM/MLLM Scoring: LLM/MLLM Scoring evaluates open-ended predictions by providing prompts, questions, standard answers, and predictions to a judging model.The passage contrasts this flexibility with option selection’s deviation from open-ended and real-world scenarios.
  • Agentic Evaluation: Agentic Evaluation uses tools to mitigate a single model’s limitations when processing diverse multimodal information.CIGEval is described as integrating a multifunctional toolbox for finer-grained evaluation.

6 Conclusion

The survey finds that current multimodal reasoning models are largely language-centric and remain weaker in visual-centric long reasoning and interactive multimodal reasoning. It therefore proposes inherently multimodal models centered on agentic reasoning, omni-modal understanding and generation, and real-world adaptability.

  • Current State: Current models predominantly use language-centric reasoning and perform strongly on visual question answering, visual math, and video understanding.The conclusion contrasts these strengths with unresolved visual-centric long reasoning and interactive multimodal reasoning.
  • Current State: Visual-centric long reasoning remains limited for tasks involving 3D contexts and complex visual information-seeking questions.The conclusion identifies this as a continuing weakness alongside interactive multimodal reasoning.
  • Future Direction: The proposed future direction is inherently large multimodal models that transcend language-dominated architectures.This concept is grounded in empirical evaluations and experimental insights.
  • Target Capabilities: Natively multimodal systems should support proactive environmental interaction, omni-modal semantic integration, and coherent cross-modal generation.Examples include embodied agents learning through trial and error, aligning vision, audio, and text, and generating diagrams or video narratives.
  • Implications: These capabilities are intended to bridge isolated task performance and generalized real-world problem-solving.The stated goal is human-like contextual adaptability across complex, open-world settings.
Loading 2505.04921v2…