PaperlayerView exact revision on arXiv

Source-linked AI summary

Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models

Yunxin Li, Zhenyu Liu, Zitao Li, Xuanyu Zhang, Zhenran Xu, Xinyu Chen, Haoyuan Shi, Shenyuan Jiang, Xintong Wang, Jifang Wang, Shouzheng Huang, Xinping Zhao, Borui Jiang, Lanqing Hong, Longyue Wang, Zhuotao Tian, Baoxing Huai, Wenhan Luo, Weihua Luo, Zheng Zhang, Baotian Hu, Min Zhang

arXiv:2505.04921v2cs.CVcs.CL

TL;DR

Multimodal reasoning models still face limited generalization, shallow reasoning, and weak agent-like behavior in complex environments. This survey organizes the field’s evolution from modular pipelines to unified multimodal reasoning and native models, concluding that future systems should support omni-modal, agentic reasoning and planning.

  • Problem

    Multimodal reasoning models still exhibit limited generalization, insufficient reasoning depth, and weak agent-like behavior, especially for complex real-world tasks.

    Source: §1 · p. 3, Stage 3: Language-Centric Long… · p. 5

  • Method

    The survey synthesizes about 700 publications through a three-stage roadmap spanning modular reasoning, multimodal chain-of-thought, and long-horizon system-2 reasoning.

    Source: §1 · p. 3, §2 · p. 4

  • Results

    The synthesis shows that current multimodal reasoning models make notable progress but consistently fall short of human-level reliability on complex, open-ended tasks.

    Source: §4.1 · p. 26

  • Takeaways & Limitations

    Future native multimodal models should prioritize agentic interaction, omni-modal understanding, and coherent multimodal generation for adaptive reasoning in open-world settings.

    Source: §6 · p. 42

    Language-centric architectures remain constrained by limited modality coverage and deficiencies in interactive, long-horizon reasoning and control.

    Source: Stage 4: Towards Native Large… · p. 5

Abstract

from arXiv · show

Reasoning lies at the heart of intelligence, shaping the ability to make decisions, draw conclusions, and generalize across domains. In artificial intelligence, as systems increasingly operate in open, uncertain, and multimodal environments, reasoning becomes essential for enabling robust and adaptive behavior. Large Multimodal Reasoning Models (LMRMs) have emerged as a promising paradigm, integrating modalities such as text, images, audio, and video to support complex reasoning capabilities and aiming to achieve comprehensive perception, precise understanding, and deep reasoning. As research advances, multimodal reasoning has rapidly evolved from modular, perception-driven pipelines to unified, language-centric frameworks that offer more coherent cross-modal understanding. While instruction tuning and reinforcement learning have improved model reasoning, significant challenges remain in omni-modal generalization, reasoning depth, and agentic behavior. To address these issues, we present a comprehensive and structured survey of multimodal reasoning research, organized around a four-stage developmental roadmap that reflects the field's shifting design philosophies and emerging capabilities. First, we review early efforts based on task-specific modules, where reasoning was implicitly embedded across stages of representation, alignment, and fusion. Next, we examine recent approaches that unify reasoning into multimodal LLMs, with advances such as Multimodal Chain-of-Thought (MCoT) and multimodal reinforcement learning enabling richer and more structured reasoning chains. Finally, drawing on empirical insights from challenging benchmarks and experimental cases of OpenAI O3 and O4-mini, we discuss the conceptual direction of native large multimodal reasoning models (N-LMRMs), which aim to support scalable, agentic, and adaptive reasoning and planning in complex, real-world environments.

1. Introduction

Large Multimodal Reasoning Models integrate text, images, audio, and video to support perception, understanding, and deep reasoning in dynamic, uncertain environments. This survey organizes the field’s rapid evolution into a structured roadmap while highlighting persistent limitations in generalization, reasoning depth, and agent-like behavior.

1 Introduction

  • LMRMs integrate text, images, audio, and video to enable logical deduction, causal inference, analogical mapping, and long-horizon thinking for decision-making across diverse environments.Reasoning supports adaptive responses, logical inferences, knowledge generalization, and navigation of complex challenges (Su et al., 2024; de Winter et al., 2024; Bi et al., 2025).Source: §1 · p. 3
  • Multimodal reasoning has progressed from perception-driven modular pipelines toward language-model-based unified multimodal understanding and reasoning, with instruction tuning and reinforcement learning improving performance.These advances bring models closer to human-like deliberative behavior (Liu et al., 2023a; DeepSeek-AI et al., 2025).Source: §1 · p. 3
  • Despite rapid progress, multimodal reasoning remains a core bottleneck because large multimodal models have limited generalization, reasoning depth, and agent-like behavior.The limitations are discussed in relation to multimodal reasoning challenges (Yue et al., 2024; Zhang et al., 2024f; Liu et al., 2024f).Source: §1 · p. 3
  • The survey addresses gaps in prior reviews by covering the roadmap from early modular designs to state-of-the-art LMRMs and projecting future developments through experimental findings and technical scrutiny.It proposes a three-stage framework beginning with Perception-Driven Modular Reasoning and Language-Centric Short Reasoning (System-1).Source: §1 · p. 3

2. Evolving Paradigms of Multimodal Reasoning and Discussion

The survey traces multimodal reasoning from modular, perception-driven systems toward language-centric reasoning and proposes N-LMRMs as a future paradigm grounded in native omnimodal perception, interaction, and goal-driven cognition. It organizes this evolution into stages while covering architectures, learning methods, datasets, benchmarks, and persistent reasoning limitations.

2. Evolving Paradigms of Multimodal Reasoning and Discussion

  • N-LMRMs are proposed as a forward-looking paradigm in which reasoning natively emerges from omnimodal perception, interaction, and goal-driven cognition rather than being retrofitted onto language models.The proposed directions emphasize unified representations, training-data synthesis, learning from world experience, and benchmark construction.Source: §2 · p. 4
  • The survey covers about 700 publications and proposes a three-stage roadmap from modular reasoning to multimodal chain-of-thought and long-horizon system-2 reasoning.It also reorganizes multimodal understanding and reasoning datasets and benchmarks, updated to 2025.06.30, by categories and evaluation dimensions.Source: §2 · p. 4

Stage 1: Perception-Driven Modular Reasoning - Designing Task-Specific Reasoning Systems

  • Early modular systems decomposed reasoning into representation, alignment, fusion, and reasoning, using CNNs and LSTMs under supervised learning before pretrained vision–language models unified earlier stages.Limited multimodal data, immature architectures, and underdeveloped learning methods motivated this task-specific design.Source: Stage 1: Perception-Driven Modular… · p. 4
  • Stage-one unification prioritized visual representation and cross-modal fusion, leaving reasoning largely implicit and classification-oriented, with limited context-aware generalization and continued reliance on task-specific modules.The passage identifies deeper language semantic modeling as the main weakness of this paradigm.Source: Stage 1: Perception-Driven Modular… · p. 4

Stage 2: Language-Centric Short Reasoning - System-1 Reasoning

  • MLLMs shifted multimodal reasoning toward end-to-end, language-centric frameworks and achieved strong performance in visual commonsense reasoning, visual question answering, and visual grounding.This stage represents the transition from modular systems to language-centric short reasoning.Source: Stage 2: Language-Centric Short… · p. 4

2. Evolving Paradigms of Multimodal Reasoning and Discussion

Multimodal reasoning has progressed from short, reactive CoT and MCoT traces toward deliberate System-2 reasoning with cross-modal chains, structured procedures, and reinforcement learning. These advances improve depth and adaptability, but language-centric LMRMs remain constrained by limited modality coverage and interactive long-horizon capabilities.

2. Evolving Paradigms of Multimodal Reasoning and Discussion

  • CoT and MCoT made multimodal reasoning more explicit and structured, but early approaches remained short, reactive, and limited on abstraction, compositionality, and planning (Kojima et al., 2022; Zhang et al., 2023g; Fei et al., 2024; Zhang et al., 2023b; Shao et al., 2024).MCoT evolved from prompt-based step-by-step traces toward decomposed reasoning paths and tool- or retrieval-augmented inference.Source: §2 · p. 5

Stage 3: Language-Centric Long Reasoning - System-2 Thinking and Planning

  • System-2-inspired LMRMs extend reasoning through cross-modal chains, longer transferable procedures, and reinforcement learning, enabling deeper reasoning, improved planning, robustness, and adaptive generalization (DeepSeek-AI et al., 2025).This transition moves multimodal reasoning from reactive behavior toward deliberative, system-level intelligence in open and dynamic environments.Source: Stage 3: Language-Centric Long… · p. 5

Stage 4: Towards Native Large Multimodal Reasoning Model (Prospect)

  • Language-centric LMRMs remain constrained by reliance on vision and language, limiting real-world integration of audio, tactile, sensor, and temporal data and weakening multimodal generation, reflection, and control (Kumar et al., 2025; Pfister & Jud, 2025).The passage also identifies deficiencies in interactive, long-horizon capabilities, but does not provide further detail.Source: Stage 4: Towards Native Large… · p. 5

3. Roadmap of Multimodal Reasoning Models

The roadmap points from modular, perception-driven systems toward native multimodal reasoners designed for unified understanding, generation, and adaptive planning. It identifies real-time interaction, agentic reasoning, and omni-modal generalization as central directions for future multimodal models.

3. Roadmap of Multimodal Reasoning Models

  • Although current models can produce extended reasoning chains in static settings, real-time iterative interaction with dynamic environments remains underdeveloped.Source: §3 · p. 6
  • Native large multimodal reasoning models (N-LMRMs) are proposed as an end-to-end paradigm unifying multimodal understanding, generation, and agentic reasoning in a shared representation space.The proposed capabilities include long-horizon planning, dynamic adaptation, embodied learning, heterogeneous data fusion, contextual multimodal generation, and modality-agnostic inference.Source: §3 · p. 6
  • The roadmap traces an evolution from modular perception-driven systems toward more unified, adaptive, and comprehensive high-level multimodal reasoning systems.Source: §3 · p. 6

3.1 Stage 1 Perception Driven Modular Reasoning - Developing Task-Specific Reasoning Modules

Stage 1 developed perception-driven multimodal reasoning through task-specific modules for representation, alignment, fusion, and inference. Transformer-based pretrained VLMs later unified these processes, but predefined alignments and implicit reasoning limited complex multi-step inference.

3.1 Stage 1 Perception Driven Modular Reasoning - Developing Task-Specific Reasoning Modules

  • Early multimodal systems used task-specific modules for representation, alignment, fusion, and reasoning because data, architectures, and learning methods were still limited.The survey distinguishes modular reasoning networks from pretrained VLM-based modular reasoning.Source: §3.1 · p. 6

3.1.1 Modular Reasoning Networks

3.1.2 Vision-Language Models-based Modular Reasoning

  • Transformer-based pretrained VLMs used large-scale image-text data to unify multimodal representation, perception, fusion, and inference across reasoning tasks.The survey groups them into dual-encoder contrastive models, single-Transformer interactive models, and multimodal LLMs using vision encoders with language models.Source: §3.1.2 · p. 8
  • Predefined feature alignments and contextual encodings often constrained iterative or compositional multi-step reasoning, motivating Multimodal Chain-of-Thought methods.Multimodal large language models nevertheless provided a more unified architecture and improved multi-task reasoning performance.Source: §3.1.2 · p. 9

3.2 Stage 2 Language-Centric Short Reasoning - System-1 Reasoning

Stage 2 develops short, language-centric multimodal reasoning through MCoT, progressing from prompt-induced chains to supervised structures and externally augmented workflows. These approaches improve interpretability, grounding, scalability, and task adaptability without requiring major architectural changes.

3.2 Stage 2 Language-Centric Short Reasoning - System-1 Reasoning

  • MCoT introduces intermediate reasoning steps that improve cross-modal alignment, knowledge integration, and contextual grounding with limited supervision or architectural modification.The stage comprises prompt-based MCoT, structural reasoning with predefined patterns, and tool-augmented reasoning with lightweight external modules.Source: §3.2 · p. 10

3.2.1 Prompt-based MCoT

  • Prompt-based MCoT extends textual chain-of-thought across modalities, enabling interpretable step-by-step reasoning with minimal additional training across visual, video, domain-specific, and task-oriented applications.Examples include visual grounding and verification, progressive video reasoning, autonomous-driving decision chains, empathetic response generation, affordance extraction, and pixel-semantic alignment.Source: §3.2.1 · p. 10

3.2.2 Structural Reasoning

  • Structural reasoning learns standardized, stage-wise multimodal workflows that improve scalability, reliability, efficiency, interpretability, and consistency through rationales, defined procedures, and modality-aware constraints.Methods construct or select rationales, separate perception from decision-making, decompose tasks into procedural phases, and use grounding, captions, sub-questions, or sensorimotor actions for modality-specific reasoning.Source: §3.2.2 · pp. 11–12

3.2.3 Externally Augmented Reasoning

  • Externally augmented reasoning offloads planning, grounding, perception, or knowledge access to search algorithms, textual tools, retrieval systems, and specialized multimodal modules.These additions support multiple reasoning paths, dynamic feedback and memory, long-horizon planning, domain-specific retrieval, and task-adaptive multimodal reasoning.Source: §3.2.3 · pp. 13, 15

3.3 Stage 3 Language-Centric Long Reasoning - System-2 Thinking and Planning

Stage 3 shifts multimodal reasoning toward deliberate, compositional System-2 processes that extend reasoning chains, ground them in multimodal inputs, and use planning or learning signals for adaptive decomposition. This stage develops through cross-modal augmentation, Multimodal-O1 generation and search strategies, and Multimodal-R1 reinforcement learning, although broad generalization remains limited.

3.3 Stage 3 Language-Centric Long Reasoning - System-2 Thinking and Planning

  • Stage 3 advances deliberate, compositional System-2 reasoning by extending multimodal chains and using explicit planning with supervised or reinforcement signals for adaptive problem decomposition.This direction addresses the shallow depth and limited adaptability of structural reasoning.Source: §3.3 · p. 15

3.3.1 Cross-Modal Reasoning

  • Cross-modal reasoning combines external tools, algorithmic control, and intrinsic multimodal capabilities to produce more grounded, interpretable, and robust inference.Program generation, procedural execution, dynamic tool orchestration, visual-rationale generation, and CoT fine-tuning support reasoning across text, images, video, and other modalities.Source: §3.3.1 · pp. 15–16

3.3.2 Multimodal-O1

  • Multimodal-O1 models deepen CoT through four-stage Summary, Caption, Thinking, and Answer generation, supported by rationale-rich fine-tuning and search methods such as Beam Search and MCTS.Best-of-N sampling, Beam Search, and MCTS expand or prioritize candidate reasoning paths to improve solution quality and long-horizon coherence.Source: §3.3.2 · pp. 17–18

3.3.3 Multimodal-R1

  • Multimodal-R1 methods use DPO and GRPO to optimize complex reasoning paths, improve multimodal capabilities across image, video, and audio tasks, and support more stable training through reward design.Despite these advances, existing approaches remain task-specific and have not yet generalized long-chain reasoning learned in mathematics to broad model capabilities.Source: §3.3.3 · p. 18

4 Towards Native Multimodal Reasoning Model

This section argues that current LMRMs remain limited in omni-modal understanding and reliable long-horizon interaction, motivating N-LMRMs that unify multimodal reasoning, generation, and agentic behavior. It outlines their capabilities and technical pathways, including unified representations, interleaved multimodal reasoning, continual learning, and data synthesis.

4.1 Experimental Findings

  • Omni-modal benchmarks reveal severe limitations: Gemini-1.5-Pro and GPT-4o average below 20% on OmniMMI, while Claude 3.5 Sonnet reaches 35% on AVQA.Performance declines further as modalities increase, indicating difficulty reasoning over image, video, and audio inputs compared with text.Source: §4.1 · p. 25, Table 12 · p. 25
  • GPT-4o achieves 0.6% accuracy on BrowseComp, rising to 1.9% with browsing, while o1 reaches 9.9% and Deep Research reaches 51.5%.These results expose persistent weaknesses in real-world grounding, long-horizon planning, tool integration, and cross-modal robustness despite progress on agent benchmarks.Source: §4.1 · p. 26
  • OpenAI o3 can decompose visual tasks through extended tool-assisted reasoning, but it misreads visual evidence, mishandles files, produces inaccurate multimedia, and fabricates rationales.In one case, o3 reaches the correct answer after 8 minutes and 13 seconds by iteratively cropping image subfigures.Source: §4.1 · pp. 26–27, Figure 6 · p. 26

4.2 Capability of N-LMRMs

  • N-LMRMs are proposed to combine Multimodal Agentic Reasoning with Omni-Modal Understanding and Generative Reasoning, capabilities that existing systems largely explore separately.The target is native integration of multimodal understanding, generation, and agentic reasoning across any modality, beyond o4-mini’s scope.Source: §4.2 · p. 27
  • Current multimodal agents show initial real-time adaptation and embodied interaction, but remain unable to sustain interactive reasoning across diverse modalities.Examples include GUI interaction, autonomous retrieval, tool-based task decomposition, and learning through virtual or physical environments.Source: §4.2 · pp. 28–29

4.3 Technical Prospects

  • N-LMRM development requires heterogeneous-modality architectures, cross-modal reasoning chains, continuous world learning, and richer synthetic data for multi-modal interaction and planning.Proposed mechanisms include unified representations or MoE, interleaved multimodal chain-of-thought with tool use, continual reinforcement learning, and data covering three or more modalities.Source: §4.3 · pp. 30–31

5. Dataset and Benchmark · 5.1 Multimodal Understanding

The survey organizes multimodal datasets and benchmarks by capacity into Understanding, Generation, Reasoning, and Planning, with Multimodal Understanding divided into visual-centric and audio-centric evaluation. Visual benchmarks span images, documents, multilingual data, and videos, while audio benchmarks cover speech, environmental sounds, and music.

5. Dataset and Benchmark

  • Datasets and benchmarks are organized into four capacity-based categories—Understanding, Generation, Reasoning, and Planning—with eleven subcategories summarized across the survey.The taxonomy is illustrated in Figure 10 and detailed in Table 14.Source: §5 · p. 32, Figure 10 · p. 32

5.1 Multimodal Understanding

  • Multimodal Understanding evaluates models’ ability to process and interpret multiple modalities for comprehension, reasoning, and generation, primarily through visual-centric and audio-centric tasks.Table 14 covers Understanding and Generation benchmarks, including visual- and audio-oriented datasets that may require short or long reasoning.Source: §5.1 · pp. 32–33, Table 14 · p. 33

5.1.1 Visual-Centric Understanding

  • Visual-centric understanding covers general visual tasks, document and chart interpretation, multilingual reasoning, video understanding, mathematical and scientific reasoning, and comprehensive evaluation.These domains range from object recognition and spatial reasoning to structured visual data interpretation and complex multimodal reasoning.Source: §5.1.1 · pp. 33, 35
  • General visual datasets have progressed from basic image VQA toward image-text alignment, object relationships, large-scale image-text representation, and compositional or spatial reasoning.Examples include VQA, GQA, ALIGN, Visual Genome, LAION-400M, LAION-5B, FILIP, YFCC100M, MMSI-Bench, WikiMixQA, and VideoMathQA.Source: §5.1.1 · pp. 33–34
  • Structured visual benchmarks require OCR-visual integration and multi-step reasoning over documents, charts, text in images, diagrams, and digital interfaces.DocVQA, DVQA, TextVQA, OCR-VQA, AI2D, and WebUIBench target document interpretation, chart understanding, text reading, graphical reasoning, or web-interface grounding and control.Source: §5.1.1 · p. 34
  • Video benchmarks evaluate temporal understanding across actions, events, and relationships, extending to scientific, long-video, and comprehensive video reasoning settings.VideoVista spans 14 video categories, 19 understanding tasks, and 8 reasoning tasks across durations from seconds to over 10 minutes.Source: §5.1.1 · p. 34

5.1.2 Audio-Centric Understanding

  • Audio-centric understanding evaluates speech, environmental sounds, and music through recognition, translation, emotion, captioning, question answering, and music-structure tasks.Comprehensive benchmarks such as VoiceBench, AudioBench, Air-Bench, MMAU, and SD-eval combine these capabilities across varied and mixed real-world audio scenarios.Source: §5.1.2 · p. 36

5.2 Multimodal Generation

Evaluations of multimodal generation assess models’ versatility and effectiveness in real-world applications, including their ability to handle complex audio data.

5.2 Multimodal Generation

  • Evaluations are crucial for advancing models’ versatility and effectiveness in real-world applications involving complex audio data.Source: §5.2 · p. 37

5.2 Multimodal Generation · 5.3 Multimodal Reasoning

The section organizes multimodal generation into cross-modal and joint multimodal tasks, emphasizing alignment, coherence, and increasingly complex transformations across modalities. It also surveys multimodal reasoning benchmarks spanning speech and robotics, alongside emerging goals for multi-step reasoning and adaptive multimodal interaction.

5.2 Multimodal Generation

  • Multimodal generation comprises cross-modal generation between modalities and joint generation across multiple modalities, with both requiring aligned, coherent outputs.Cross-modal tasks generate one modality from another, whereas joint tasks simultaneously generate multiple modalities.Source: §5.2 · p. 37

5.2.1 Cross-modal Generation

  • Cross-modal generation evaluates transformations such as text-to-image, text-to-video, and text-to-speech, focusing on alignment, coherence, and semantic generation under complex or conditional prompts.These tasks require mapping information between modalities while preserving meaning and handling increasingly sophisticated inputs.Source: §5.2.1 · p. 37, §5.3 · p. 38
  • Text-to-image research uses datasets and benchmarks to assess general generation, editing, conditional control, compositional prompting, object interactions, instruction completeness, and semantic alignment.The surveyed resources range from MSCOCO (30K), CC12M, and Flickr30k to GenEval, T2I-CompBench++, MagicBrush, and UniCombine.Source: §5.2.1 · p. 37
  • Text-to-video benchmarks evaluate relevance, coherence, visual quality, user satisfaction, and real-world accuracy across broad and specialized video-generation settings.Datasets include VidGen-1M, OpenVid-1M, and VidProM, while benchmarks include AIGCBench, EvalCrafter, VBench, VideoScore, WorldSimBench, and WorldScore.Source: §5.2.1 · p. 37

5.3 Multimodal Reasoning

  • Speech-oriented multimodal reasoning benchmarks compare text and audio outputs using word error rate and accuracy, while newer evaluations cover multilingual, ambiguous, professional, stylistic, emotional, and intonational speech scenarios.ADU-Bench and URO-Bench extend evaluation beyond basic speech-dialogue capability.Source: §5.3 · p. 38
  • Robotics benchmarks provide interactive, high-fidelity multimodal environments for tasks such as autonomous driving and standardized evaluation across diverse robotics capabilities.ThreeDWorld and GAIA-1 provide simulation platforms, while Genesis supplies a standardized evaluation framework.Source: §5.3 · p. 38

5.2.2 Joint Multimodal Generation

  • Joint multimodal generation creates multiple modalities simultaneously, requiring intermodal coherence and alignment in outputs such as interleaved image-text sequences and text-generated multimodal stories.MM-Interleaved, ANOLE, InterleavedEval, OpenLEAF, and OpenING support interleaved generation, while NextGPT, DreamFactory, and EVA extend text-to-multimodal output using multimodal stories, video evaluation, and embodied world models.Source: §5.2.2 · p. 38

5.3 Multimodal Reasoning · 5.3 Multimodal Reasoning

Multimodal reasoning requires integrating information across modalities to infer, solve problems, and answer complex questions. The section distinguishes general visual reasoning from domain-specific reasoning and surveys benchmarks and datasets supporting both capabilities.

5.3 Multimodal Reasoning

  • Multimodal reasoning integrates multiple modalities to support inference, problem-solving, and complex question answering beyond simple understanding or generation.Source: §5.3 · p. 38
  • The field broadly separates general visual reasoning from domain-specific reasoning, including technical abilities such as mathematical problem-solving from visual inputs.Source: §5.3 · p. 38

5.3.1 General Visual Reasoning

  • General visual reasoning requires perceiving, comprehending, analysing, and reasoning about visual information using knowledge, logic, and common sense across varied scenarios.Source: §5.3.1 · p. 39
  • Benchmarks such as VCR (Zellers et al., 2019), PhysBench (Chow et al., 2025b), and VideoPhy (Bansal et al., 2024) test visual commonsense and physical reasoning beyond VQA.Source: §5.3.1 · p. 39
  • Broader evaluations span multilingual, multidisciplinary, human-centric, and video-centric reasoning through MMBench (Liu et al., 2024f), MMMU (Yue et al., 2024), AGIEval (Zhong et al., 2024b), VideoVista (Li et al., 2024k), and MMStar (Chen et al., 2024f).AI2D (Kembhavi et al., 2016) and InfographicVQA target diagrams, charts, infographics, spatial layouts, and relationships.Source: §5.3.1 · p. 39
  • Training resources include SWAG (Zellers et al., 2018), LLava-CoT (Xu et al., 2024b), CLEVR (Johnson et al., 2016), Mulberry-260K (Yao et al., 2024a), ShareGPT4oReasoning (Zhang et al., 2024e), Video-R1-data (Feng et al., 2025b), and Visual-CoT (Shao et al., 2024).These datasets support action prediction, visual commonsense, synthetic-image reasoning, multimodal dialogue, dynamic-video reasoning, and visual chain-of-thought training.Source: §5.3.1 · p. 39

5.3.2 Domain-specific Reasoning

5.4 Multimodal Planning · 5.4 Multimodal Planning

Multimodal planning benchmarks evaluate agents that integrate visual, textual, and interactive inputs across complex multi-step tasks. They span GUI navigation and embodied or simulated environments, testing planning, reasoning, adaptability, and generalization.

5.4 Multimodal Planning

  • Domain-specific reasoning benchmarks target specialized applications in mathematics, robotics, and physical laws, while broader systems aim to generalize across tasks and real-world scenarios.These benchmarks push multimodal reasoning toward more adaptive, intelligent, and versatile systems.Source: §5.4 · p. 40
  • Planning benchmarks assess multimodal agents across web navigation, GUIs, embodied environments, and open-ended simulations, providing a comprehensive view of their capabilities.They evaluate integration of diverse inputs during complex, multi-step tasks.Source: §5.4 · p. 40

5.4.1 GUI Navigation

  • GUI benchmarks test visual-language grounding and multi-step execution in realistic websites, desktops, mobile interfaces, and simulated e-commerce environments.WebArena, Mind2Web, Windows Agent Arena, WebShop, OSWorld, OmniACT, and LlamaTouch cover navigation, extraction, cross-application workflows, and automation.Source: §5.4.1 · p. 40
  • VisualWebBench introduces 1.5K visual-intensive tasks for cross-page integration and element localisation, while Mind2Web tests generalizability through cross-website tasks.These benchmarks emphasize visual grounding and transfer across interfaces.Source: §5.4.1 · p. 40
  • VisualAgentBench unifies evaluation across GUI, embodied, and visual-design tasks, extending planning assessment to visually rich digital environments.It systematically evaluates large multimodal models for planning and acting.Source: §5.4.1 · p. 40

5.4.2 Embodied and Simulated Environments

  • Embodied and simulated benchmarks evaluate adaptive planning in dynamic physical or virtual worlds through navigation, manipulation, resource management, and human-robot collaboration.MineDojo, MuEP, BEHAVIOR-1K, Habitat 3.0, SAPIEN, HomeRobot, OpenVocabManip, and HoloAssist cover these capabilities.Source: §5.4.2 · p. 40
  • Specialized environments support lifelong learning, open-vocabulary mobile manipulation, real-world collaborative planning, and dynamic driving decisions using multimodal data.GVCCI achieves performance gains without human supervision, while DrivingDojo evaluates real-time multi-agent decision-making.Source: §5.4.2 · p. 40

5.5 Evaluation Method · 5.5 Evaluation Method

Evaluation of multimodal planning and reasoning uses four mainstream paradigms, while existing benchmarks still inadequately capture long-horizon, noisy, adaptive, and collaborative real-world behavior.

5.5 Evaluation Method

  • Planning benchmarks span web navigation and embodied environments, but future evaluations should target open-world settings, real-time human feedback, and multi-agent or human-AI collaboration.Current gaps include long-horizon planning, noisy inputs, and real-world adaptability; positioning, trajectory tracking, and visual memory provide ways to quantify planning abilities.Source: §5.5 · p. 41
  • The mainstream evaluation paradigms are Exact/Fuzzy Match, Option Matching, LLM/MLLM Scoring, and Agentic Evaluation.Source: §5.5 · p. 41
  • Exact or fuzzy matching evaluates open-ended VQA by comparing rule-processed predictions with human answers, using scoring rules such as VQAv2’s 1/3-point increments and DocVQA’s Levenshtein distance.Source: §5.5 · p. 41
  • Option Matching presents candidate answers for selection, while CircularEval in MMBench (Liu et al., 2024f) reduces stochastic preference for particular options.Source: §5.5 · p. 41
  • LLM/MLLM scoring evaluates open-ended predictions by prompting models such as GPT-4o with questions, references, predictions, guidelines, and examples to produce fairer scores (Fu et al., 2024b; Zhang et al., 2023f).Source: §5.5 · p. 41
  • Agentic Evaluation uses tools, cooperative or adversarial multi-agent interactions, and consensus to produce more fine-grained, reliable, and interpretable assessments.CIGEval (Wang et al., 2025n) expands visual understanding through a multifunctional toolbox, while multi-agent evaluation builds on consensus and robust solutions (Xu et al., 2023b; Chen et al., 2024d; Xu et al., 2025i; Chan et al., 2024; Li et al., 2024h; Zhao et al., 2024b; Liang et al., 2024).Source: §5.5 · p. 41

6 Conclusion

The survey concludes that language-centric multimodal reasoning has achieved strong task performance, but visual-centric long reasoning and interactive reasoning remain underdeveloped. It proposes inherently large multimodal models centered on agentic interaction, omni-modal understanding and generation, and human-like contextual adaptability.

6 Conclusion

  • Current language-centric models perform impressively on visual question answering, visual math, and video understanding, but visual-centric long reasoning and interactive multimodal reasoning remain underdeveloped.The open challenges include understanding 3D contexts, answering complex visual information-seeking questions, and supporting dynamic cross-modal dialogue or iterative feedback loops.Source: §6 · p. 41

6. Conclusion

  • Future models should combine multimodal agentic reasoning, omni-modal understanding and generative reasoning, and context-aware cross-modal outputs to support generalized real-world problem-solving.These capabilities include proactive environmental interaction, aligning semantics across vision, audio, and text, resolving open-world ambiguities, and generating outputs across modalities.Source: §6 · p. 42

Related papers

Loading 2505.04921v2…