Source-linked AI summary
Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey
Yaoting Wang, Shengqiong Wu, Yuecheng Zhang, Shuicheng Yan, Ziwei Liu, Jiebo Luo, Hao Fei
TL;DR
MCoT research has expanded across modalities and applications, but the field lacks an up-to-date comprehensive survey and still faces unresolved reasoning challenges. This paper provides the first systematic survey of MCoT, organizing foundations, methods, resources, applications, and future directions. It highlights broad research successes while identifying long-chain evaluation and multimodal error propagation as open problems.
Problem
MCoT lacks a comprehensive up-to-date survey despite growing research, unresolved multimodal reasoning questions, and expanding applications.
Method
The paper conducts the first systematic survey of MCoT, covering foundations, taxonomies, methodologies, applications, datasets, benchmarks, challenges, and future directions.
Results
The survey consolidates MCoT research across modalities and highlights successes in robotics, healthcare, autonomous driving, social science, and multimodal generation.
Takeaways & Limitations
The survey organizes the MCoT landscape and identifies future research directions toward multimodal AGI.
Takeaways & Limitations
Long MCoT chains remain vulnerable to error snowballing, while existing confidence calibration struggles with contradictions across modalities.
Abstract
from arXiv · showhide
By extending the advantage of chain-of-thought (CoT) reasoning in human-like step-by-step processes to multimodal contexts, multimodal CoT (MCoT) reasoning has recently garnered significant research attention, especially in the integration with multimodal large language models (MLLMs). Existing MCoT studies design various methodologies and innovative reasoning paradigms to address the unique challenges of image, video, speech, audio, 3D, and structured data across different modalities, achieving extensive success in applications such as robotics, healthcare, autonomous driving, and multimodal generation. However, MCoT still presents distinct challenges and opportunities that require further focus to ensure consistent thriving in this field, where, unfortunately, an up-to-date review of this domain is lacking. To bridge this gap, we present the first systematic survey of MCoT reasoning, elucidating the relevant foundational concepts and definitions. We offer a comprehensive taxonomy and an in-depth analysis of current methodologies from diverse perspectives across various application scenarios. Furthermore, we provide insights into existing challenges and future research directions, aiming to foster innovation toward multimodal AGI.
1 Introduction
MCoT extends stepwise CoT reasoning to multimodal contexts, where heterogeneous data interactions create new reasoning paradigms and unresolved design challenges. This survey addresses the lack of a comprehensive review through a taxonomy, methodological analysis, resource compilation, and discussion of future directions.
- Motivation: MCoT extends CoT reasoning to multimodal contexts, integrating diverse modalities into language intelligence and supporting intermediate reasoning steps.CoT decomposes complex problems into sequential intermediate steps, while MCoT applies this paradigm to multimodal settings.
- Challenges: Growing MCoT research must address how to leverage varied multimodal context, design effective CoT processes, and implement implicit reasoning in MLLMs.The survey identifies these as critical unanswered questions in the emerging field.
- Scope and Contributions: This paper provides the first systematic survey of MCoT reasoning, including foundational concepts, a comprehensive taxonomy, methodological analysis, applications, datasets, and benchmarks.Its organization spans background, multimodal methods, taxonomy, applications, resources, and future directions.
- Scope and Contributions: The survey compiles relevant resources and outlines challenges and future research directions intended to support progress toward multimodal AGI.Resource sharing and future-direction analysis are presented as explicit contributions.
2 Background and Preliminary
The background defines MCoT as CoT reasoning extended to multimodal components and distinguishes text-only from multimodal thoughts. It also reviews reasoning topologies and the comprehension and generation capabilities of MLLMs.
- From CoT to MCoT: CoT decomposes complex tasks into manageable subtasks, with intermediate reasoning trajectories called rationales; MCoT extends this paradigm to diverse data modalities.The paper distinguishes conventional in-context learning from rationale-generating chain-of-thought reasoning.
- From CoT to MCoT: MCoT allows multimodal information in the prompt, query, answer, or rationale, without requiring all components to be multimodal simultaneously.The framework separates language-based input T from language-excluded multimodal context M.
- From CoT to MCoT: Scenario-1 uses text-only rationales for multimodal inputs or outputs, whereas Scenario-2 incorporates given, retrieved, or generated multimodal information into the rationale.The scenarios classify MCoT according to the composition of the thought process.
- Thought Paradigms: Chain, tree, graph, and hypergraph topologies progress from sequential reasoning to branching, aggregation, and joint integration across modalities.Tree structures support exploration and backtracking, while graphs support multiple parents and aggregation; hypergraphs connect more than two thoughts.
- Multimodal LLMs: MLLM research covers both multimodal comprehension and multimodal generation, with architectures differing according to whether they only understand content or also generate it.The background reviews representative systems for image-text understanding and image, audio, and video generation.
- Multimodal LLMs: Long-MCoT strategies have demonstrated robust multimodal reasoning performance, motivating continued interest in deliberate processing and test-time scaling.The cited models include Mulberry, AStar, and LlamaV-o1.
3 MCoT Reasoning Under Various Modalities
MCoT extends chained reasoning across image, video, 3D, speech, audio, and structured-data tasks. Methods generate rationales, decompose complex inputs, and support applications including question answering, generation, and specialized analysis.
- Overview: MCoT extends reasoning across images, videos, audio, 3D scenes, tables, and charts.The survey emphasizes multimodal reasoning across heterogeneous modalities and structured data.
- Image: Image-based MCoT commonly generates intermediate rationales before final predictions and extends beyond VQA to detection, segmentation, and image generation.Methods also refine prompts, reward mechanisms, and layouts before synthesis.
- Video: Video MCoT decomposes comprehension into stages such as target identification, tracking, action analysis, question answering, and verification.Other approaches separate captioning, saliency inference, and bounding-box production for video generation.
- Specialized Applications: MCoT’s video applications include medical video analysis and affective computing, while broader video methods aim to improve accuracy and computational efficiency.The survey presents these applications as evidence of MCoT’s utility in specialized domains.
- 3D: MCoT addresses 3D tasks by decomposing shape generation, grounding, image generation, and robot-skill learning into structured processes.Chain-of-3D-Thought uses iterative trial-and-error and tool invocation, while 3D-CoT incorporates structural reasoning annotations.
- Audio and Speech: Speech and audio methods separate recognition, translation, emotion detection, and dialogue generation into manageable reasoning stages.Audio-CoT applies vanilla CoT to audio understanding and reasoning tasks.
4 Methodologies in MCoT Reasoning
The survey organizes MCoT methodologies by rationale construction, structural reasoning, information enhancement, objective granularity, multimodal rationale, and test-time scaling. These perspectives distinguish how rationales are produced, structured, enriched, and optimized.
- Taxonomy: MCoT methods are categorized by rationale construction, structural reasoning, information enhancement, objective granularity, multimodal rationale, and test-time scaling.The taxonomy provides a systematic analysis of methods from multiple perspectives.
- Rationale Construction: Rationale construction methods comprise prompt-based, plan-based, and learning-based approaches.Prompting guides inference, planning explores and refines thoughts, and learning embeds rationale construction in training or fine-tuning.
- Structural Reasoning: Structural reasoning methods use asynchronous modality modeling, defined procedure staging, or autonomous procedure staging.These formats are intended to improve the controllability and interpretability of rationale generation.
- Information Enhancing: Information-enhancing methods use expert tools, external knowledge retrieval, and in-context knowledge retrieval.Examples include visualizations, image manipulation, retrieval-augmented generation, and prioritization of relevant image regions.
- Objective Granularity: MCoT methodology often follows objective granularity, distinguishing coarse overview understanding from fine-grained instance-level grounding.Question answering generally emphasizes overview information, whereas grounding emphasizes individual instances.
- Test-Time Scaling: Reinforcement learning extends MCoT reasoning to video, detection, segmentation, and multimodal emotion recognition while supporting reflection and backtracking.The survey reports “aha-moments” in visual reasoning and describes RL as enabling iterative self-improvement without SFT.
5 Applications with MCoT Reasoning
MCoT applications span embodied systems, agents, autonomous driving, healthcare, affective computing, and multimodal generation. Across these domains, chained reasoning is used to decompose tasks, guide decisions, and structure generation or analysis.
- Embodied Systems: Embodied systems use MCoT to segment planning, manipulation, navigation, and long-horizon activities into actionable or visual subgoals.Examples include EmbodiedGPT, E-CoT, ManipLLM, CoTDiffusion, Emma-X, and SpatialCoT.
- Agents: Multimodal agents apply chained reasoning to GUI manipulation, navigation, recommendations, and iterative goal-directed self-reflection.Auto-GUI uses Multimodal Chain-of-Action, while SmartAgent uses Chain-of-User-Thought for personalized recommendations.
- Autonomous Driving: Autonomous-driving methods integrate MCoT for perception, knowledge, reasoning, decision-making, and adaptation in dynamic environments.DriveCoT uses a tailored dataset, while PKRD-CoT applies zero-shot MCoT prompting.
- Healthcare: Healthcare applications use chained reasoning for stress detection, procedural-error identification, and medical visual question answering.Reported systems emphasize interpretability and precision across medical AI tasks.
- Affective Computing: Affective-computing applications use structured rationales for empathetic dialogue and multimodal sentiment analysis.MM-PEAR-CoT generates textual rationales before late-stage multimodal fusion.
- Multimodal Generation: Image and 3D generation methods use iterative reasoning, reflection, semantic analysis, spatial analysis, and layout construction before synthesis.These strategies are presented as diverse multimodal synthesis approaches.
6 MCoT Datasets and Benchmarks
MCoT resources include rationale-based datasets for fine-tuning and benchmarks for evaluating downstream capabilities. The surveyed benchmarks cover general, mathematical, scientific, multimodal, agentic, grounding, segmentation, and hallucination-related tasks.
- Resource Organization: MCoT resources are divided into datasets for rationale-based MLLM fine-tuning and benchmarks for downstream capability evaluation.The survey organizes these resources as complementary training and assessment infrastructure.
- Rationale-Based Datasets: ScienceQA, A-OKVQA, and T-SciQ provide multimodal questions, explanations, commonsense or world knowledge, and enhanced reasoning rationales.MAmmoTH-VL-Instruct contains 12 million long-MCoT examples across 118 datasets and 10 categories.
- Resource Taxonomy: Table 3 catalogs datasets and benchmarks by answer format and modality, using MC and Open for formats and T, I, V, and A for modalities.The table’s abbreviations distinguish multiple-choice versus open-ended answers and text, image, video, versus audio.
- Downstream Benchmarks: Downstream benchmarks assess multidisciplinary, mathematical, scientific, perception, knowledge, reasoning, and agent capabilities.Examples include MMMU, MathVista, Math-Vision, EMMA, OSWorld, and AgentClinic.
- Specialized Evaluation: Specialized benchmarks evaluate grounding, spatial reasoning, video segmentation, audiovisual grounding, and hallucinations across multimodal settings.HallusionBench, AVTrustBench, and AVHBench target hallucination phenomena, while MeViS evaluates video segmentation with motion expressions.
- Rationale-Based Benchmarks: Rationale-based benchmarks evaluate multimodal outputs, long-MCoT reasoning, reasoning-step quality, and advanced mathematics and physics reasoning.CoMT, WorldQA, MiCEval, OlympiadBench, and MME-CoT assess different aspects of rationale quality and multimodal reasoning.
7 Limitations, Challenges and Future Directions
The survey identifies unresolved challenges spanning data, modality integration, reasoning reliability, embodiment, interpretability, safety, efficiency, generalization, and adaptation. It points to future work on better data strategies, open-ended evaluation, adaptive and efficient reasoning, cross-modal verification, and tighter symbolic-neural and embodied integration.
- Computational Sustainability and Slow-thinking Paradox: Long-MCoT reasoning remains bottlenecked by the exponential computational and training-data costs of test-time scaling and slow thinking.The survey highlights algorithmic efficiency, reinforcement learning, and hardware acceleration as potential directions.
- Lack of Reasoning in General Scenarios: Current long-MCoT frameworks are strongest on verifiable mathematics and science but remain less effective for ambiguous, open-ended general scenarios.The survey proposes open-ended reward models for robust multimodal reasoning in such settings.
- Error Propagation in Extended Reasoning Chains: Small early inaccuracies can snowball into catastrophic outcomes, while multimodal contradictions may persist despite high self-consistency scores.Quantitative metrics for diagnosing, measuring, and mitigating cumulative errors remain unresolved.
- Dynamic Environment Adaptation and Adaptive Chain Length: Most systems assume static inputs and cannot revise earlier conclusions from streaming evidence without restarting the reasoning chain.Adaptive chain lengths and real-time feedback are proposed to balance reasoning accuracy with computational efficiency.
- Hallucination Prevention: Hallucinations undermine MCoT reliability because multimodal integration can create misaligned contexts and spurious information.Suggested directions include cross-modal alignment, stepwise verification, uncertainty quantification, adversarial training, and external knowledge bases.
- Data Selection, Annotation, and Augmentation Strategies: Automating the selection and annotation of data suitable for extended reasoning remains an open challenge despite evidence that curated datasets can activate long-MCoT capabilities.Semi-supervised, self-supervised, and reinforcement-learning strategies may reduce reliance on extensive manual annotation.
- Modality Imbalance and High-Dimensional Modal Integration: Progress is uneven across modalities, with text and images advancing faster than higher-dimensional inputs such as 3D data and sensor information.The survey calls for integration strategies that preserve each modality’s distinctive characteristics.
- Embodied Reasoning Limitations: MCoT systems remain constrained by abstract symbol spaces, opaque theoretical mechanisms, safety risks, and incomplete symbolic-neural interfaces.The survey connects these gaps to embodiment, interpretability, ethical robustness, and crossmodal symbolic manipulation.
8 Conclusion
This survey provides the first systematic review of MCoT reasoning, covering its foundations, methods, modalities, resources, applications, and future directions. It consolidates the field’s landscape and shares resources to support follow-up research toward multimodal AGI.
- The survey presents the first systematic review of multimodal chain-of-thought reasoning.
- It defines foundational concepts and proposes a taxonomy covering MCoT approaches and reasoning paradigms across image, video, speech, audio, 3D, and structured data.
- The review consolidates datasets and benchmarks and examines applications including robotics, healthcare, autonomous driving, social science, and multimodal generation.
- It outlines future research directions addressing current limitations and shares related resources to facilitate follow-up work toward multimodal AGI.