Source-linked AI summary
Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning
Yiqi Wang, Wentao Chen, Xiaotian Han, Xudong Lin, Haiteng Zhao, Yongfei Liu, Bohan Zhai, Jianbo Yuan, Quanzeng You, Hongxia Yang
TL;DR
MLLM reasoning abilities have not been systematically investigated despite rapid progress and varied benchmark evaluations. This survey synthesizes reasoning types, models, data, evaluation protocols, and reasoning-intensive applications, then discusses current practices and future directions. It concludes by organizing the field’s current standing and identifying directions concerning pretraining, alignment, and evaluation benchmarks.
Problem
The reasoning abilities of MLLMs have not been systematically investigated, despite their importance to progress toward Strong AI or AGI.
Method
The survey reviews multimodal reasoning evaluation protocols, categorizes MLLM models and reasoning types, and summarizes applications, practices, and future directions.
Results
The survey summarizes current MLLM reasoning capabilities across benchmark results, including differences associated with underlying language models, retained language knowledge, and training choices.
Takeaways & Limitations
The survey provides a structured account of the field and outlines research directions involving pretraining, alignment, and evaluation benchmarks.
Takeaways & Limitations
Most existing reasoning-related multimodal benchmarks are not reasoning-focused, and multimodal hallucinations remain difficult to diagnose because their sources may lie in perception or language-model biases.
Abstract
from arXiv · showhide
Strong Artificial Intelligence (Strong AI) or Artificial General Intelligence (AGI) with abstract reasoning ability is the goal of next-generation AI. Recent advancements in Large Language Models (LLMs), along with the emerging field of Multimodal Large Language Models (MLLMs), have demonstrated impressive capabilities across a wide range of multimodal tasks and applications. Particularly, various MLLMs, each with distinct model architectures, training data, and training stages, have been evaluated across a broad range of MLLM benchmarks. These studies have, to varying degrees, revealed different aspects of the current capabilities of MLLMs. However, the reasoning abilities of MLLMs have not been systematically investigated. In this survey, we comprehensively review the existing evaluation protocols of multimodal reasoning, categorize and illustrate the frontiers of MLLMs, introduce recent trends in applications of MLLMs on reasoning-intensive tasks, and finally discuss current practices and future directions. We believe our survey establishes a solid base and sheds light on this important topic, multimodal reasoning.
1 Introduction
MLLMs have advanced across multimodal applications, but whether they can genuinely reason across modalities remains unresolved. This motivates a systematic review of their reasoning abilities, evaluation protocols, models, and future directions.
- MLLMs have achieved substantial progress across domains and applications as computational power and datasets have expanded.
- Multimodal reasoning requires comprehending and integrating information from modalities such as vision, text, and audio.
- Despite existing benchmark datasets, it remains too early to conclude that current MLLMs can truly reason over multiple input modalities.
- Known weaknesses in mathematical and multi-hop reasoning, together with hallucinations, can prevent LLMs and MLLMs from reasoning properly.
2 Reasoning: Definition and Evaluation Protocols
The survey defines reasoning through premises, inferences, and conclusions, then focuses on informal deductive, abductive, and analogical reasoning in MLLMs. These forms combine logical processing with domain knowledge and support practical multimodal reasoning analysis.
- What is Reasoning?: Reasoning integrates premises, intermediate inferences, and conclusions to derive new claims from given information.Inferences select and interpret information, make connections, verify it, and draw conclusions.
- What is Reasoning?: Valid practical reasoning requires both logical rules and domain knowledge, such as mathematics for arithmetic or commonsense for everyday tasks.
- What is Reasoning?: Formal reasoning guarantees true conclusions when premises are true, whereas informal reasoning does not guarantee truth with incomplete or ambiguous information.
- Reasoning Types: Deductive reasoning derives conclusions step by step from known premises according to logical rules, even when the premises themselves are wrong.
- Reasoning Types: Analogical reasoning transfers knowledge from similar instances to a new case by inferring and applying a hidden proposition.
- Reasoning Types: The survey studies MLLM reasoning as informal reasoning and focuses on deductive, abductive, and analogical reasoning because of their real-world relevance.
2.2 Language-only Reasoning Tasks
Language-only reasoning tasks cover mathematical, commonsense, symbolic, and environment-interaction settings. They test operations ranging from arithmetic and everyday knowledge to rule-based manipulation, planning, feedback processing, and action adjustment.
- Language-only reasoning tasks require no images, unlike multimodal reasoning tasks that involve both images and text.
- Mathematical Reasoning: Mathematical reasoning derives answers through one-step or multistep arithmetic operations using question understanding and conceptual knowledge.
- Commonsense Reasoning: Commonsense reasoning uses broadly shared knowledge across social, physical, and biological domains for everyday decision-making.
- Symbolic Reasoning: Symbolic reasoning manipulates abstract objects under precisely defined rules, including proving formulas from facts and logical rules.
- Environment Interaction: Environment-interaction tasks require understanding situations, planning actions, processing feedback, and adjusting subsequent actions.
2.3 Multimodal Reasoning Benchmarks
The survey argues that ideal multimodal reasoning benchmarks must require multimodal information, distinguish reasoning types, and annotate reasoning steps. It reviews benchmark datasets and metrics across capability-based and open- versus closed-set evaluations, while noting that most existing benchmarks are not reasoning-focused.
- Benchmark Datasets: An ideal multimodal reasoning benchmark requires multimodal information, follows reasoning categories, and provides detailed annotations of reasoning steps.
- Benchmark Datasets: Traditional captioning and visual-question-answering datasets provide initial platforms for evaluating MLLM comprehension and reasoning capabilities.
- Benchmark Datasets: Most existing reasoning-related multimodal benchmarks are not reasoning-focused; MM-Vet lacks reasoning categorization and steps, while MMMU has only partial step annotations.
- Evaluation Metrics: Multimodal evaluation metrics must assess prediction accuracy together with the depth, nuance, and relevance of associations across modalities.
- Evaluation Metrics: Evaluation metrics cover vision, language, and reasoning capabilities, using measures such as mAP, BLEU, QA-Accuracy, Elo score, and GPT-4 evaluation.
- Evaluation Question Types: Benchmarks also differ by question format: closed-set evaluations restrict possible responses, whereas open-set evaluations allow unbounded generation and test generalization.
- Evaluation Question Types: The survey’s analysis uses Infi-MM-Eval, MM-Vet, and MMMU, covering GPT-4 evaluation, QA-Accuracy, open-set questions, and closed-set questions.
2.4 Improving Reasoning Abilities for LLMs
Research improves LLM reasoning through supervised training, in-context learning, prompt engineering, and external tool use. These approaches have also been explored for multimodal reasoning in MLLMs.
- Supervised pretraining or fine-tuning uses mathematical corpora or instruction-tuning data to improve LLM reasoning.Mathematical corpora target quantitative reasoning, while instruction-tuning data fine-tunes pretrained models.
- In-context learning and prompt engineering, including Chain-of-Thought methods, can improve reasoning abilities.Chain-of-Thought and related variants are presented as prompt-based approaches for reasoning improvement.
- External tools such as Python code interpreters can improve the accuracy of individual reasoning steps.
- Instruction tuning, prompt engineering, and tool usage have also been explored for multimodal reasoning in MLLMs.
3 Multimodal Large Language Models
MLLMs combine visual encoders, language models, and connectors, with differing interaction modules, connector architectures, training data, and training stages. Recent models show expanding benchmark performance through instruction tuning and increasingly complex training frameworks.
- Architecture: MLLMs typically combine a visual encoder and language model through a connector, with both components commonly initialized from pretrained models.
- Architecture: Some MLLMs inject visual tokens directly into the language model input, whereas others use cross-attention between visual and language features.LLaVA directly inputs visual signals, while Flamingo uses cross-attention layers within transformer blocks.
- Architecture: Connector designs include query-based modules such as Q-Former or perceiver resampler and MLP connectors.BLIP-2, Flamingo, and QWen-VL use query-based connectors, while LLaVA and MiniGPT4-v2 use MLPs.
- Model development: Flamingo pioneered query-based cross-attention through the perceiver resampler, while BLIP-2 introduced the Q-Former to bridge vision and language.The Q-Former was initialized from pretrained BERT and serves as a query-based sampler.
- Training and results: InstructBLIP and LLaVA-1.5 improve benchmark performance through instruction tuning, with LLaVA-1.5 additionally integrating VQA datasets.
- Training and results: QWen-VL and MiniGPT-v2 use three-stage training frameworks, while InstructBLIP and LLaVA-1.5 use pretraining followed by instruction tuning.QWen-VL adds multi-task training with high-quality supervised data; MiniGPT-v2 emphasizes task-specific instruction templates.
4 Multimodal Reasoning through Instruction Tuning
Multimodal instruction tuning extends instruction-following and in-context learning to inputs that combine modalities with language. The section surveys dataset design, ICL preservation challenges, representation learning, exemplar generation, and model interaction as routes to multimodal reasoning.
- 4 Multimodal Reasoning through Instruction Tuning: Multimodal instruction tuning adapts pretrained models to tasks requiring comprehension and integration across modalities.
- 4.1 Definition of Instruction Tuning: Instruction tuning fine-tunes models on instruction-response pairs, with supervision computed solely from response tokens.
- 4.1 Definition of Instruction Tuning: Multimodal instruction-tuning examples combine an X-modality such as images, videos, or audio with an instruction and response.
- 4.2 Instruction Tuning for In-Context Learning: MLLMs demonstrate in-context learning, enabling adaptation to new tasks from a few examples without updating model weights.
- 4.2 Instruction Tuning for In-Context Learning: Instruction tuning focused mainly on image-text pairs can diminish in-context learning capabilities because ICL-focused datasets are difficult to construct.
- 4.3 Multimodal Prompting by Representation Learning: Multimodal prompting seeks LLM-compatible representations of non-linguistic inputs while avoiding changes to the base language model.
- 4.4 Multimodal Prompting by Exemplar Generation: Img2LLM automatically generates visual question-answer exemplars by extracting answer candidates from captions and formulating corresponding questions.
- 4.5 Multimodal Prompting by Model Interaction: Interactive multimodal prompting uses language as a bridge between foundation models, allowing zero-shot task performance without additional training.
5 Applications
The applications section introduces Embodied AI and multimodal agents as application areas for MLLMs.
- The section discusses Embodied AI and multimodal agents as applications of MLLM models.
5.1 Embodied AI
Embodied AI uses MLLMs and related pretrained models to perceive environments, reason about tasks, plan actions, and incorporate feedback. Applications span navigation, manipulation, question answering, world modeling, policy generation, and robotic foundation models.
- 5.1 Embodied AI: Embodied AI agents must see, talk, listen, interact with environments, and reason about actions using environmental cues and feedback.
- 5.1.1 MLLM for Embodied AI: MLLMs support robotic manipulation by combining visual commonsense knowledge with language-model reasoning for planning, code generation, and control commands.
- 5.1.1 Direct Reasoning with MLLM Knowledge: Early embodied approaches provide agent observations to pretrained models, which decompose tasks into smaller executable steps for planning and problem-solving.
- 5.1.2 Reasoning with Separate Components: Separate-component systems combine pretrained models through language prompts, including CLIP-based semantic understanding and GPT-3-CLIP navigation pipelines without further fine-tuning.
- 5.1.2 Reasoning with External World Model: World models record key environmental states to extend reasoning beyond the language model’s context window, while feedback systems update prompts with textual observations.
- 5.1.3 Reasoning with Policy Generation: Policy-generation approaches use language-model programming to produce code or policies for robotic control, but direct code-execution systems remain constrained by available APIs and control primitives.
- 5.1.3 Improved Reasoning with Robotic Data: DIAL expands human instruction candidates with GPT-3 and reports improved capabilities for solving novel tasks.
- 5.1.4 Embodied Foundation Model: RT-2 achieves better generalization than robotics-only models by fine-tuning with both robotics data and original web data.
5.2 MLLMs for Tool Usage
MLLMs can serve as tools within larger reasoning systems, coordinating modality-specific models, external APIs, and programs for complex visual, dialog, mathematical, and multimodal tasks.
- 5.2 MLLMs for Tool Usage: Tool-augmented language models select or create tools and process their outputs to address tasks beyond their inherent capabilities.
- 5.2.1 Tool Learning for LLMs: Complex tool-using tasks are commonly decomposed into sequential or tree-structured subtasks using methods such as Chain of Thoughts, Tree of Thoughts, and Program of Thoughts.
- 5.2.1 Tool Learning for LLMs: Models learn tool use through descriptions, few-shot examples, or fine-tuning on tool-specific corpora, including self-supervised API-use data.
- 5.2.1 Tool Learning for LLMs: Gorilla’s APIBench-trained LLaMA-based model significantly surpasses GPT-4 in generating API calls.
- 5.2.1 Tool Learning for LLMs: Program-generating approaches use programming languages for mathematical computation and logical task planning.
- 5.2.2 MLLMs as Tool: VISPROG uses MLLMs to generate programs that coordinate specialized vision modules for compositional VQA, image-pair reasoning, object tagging, and image editing.
- 5.2.2 MLLMs as Tool: Visual ChatGPT and MM-ReAct combine MLLMs or visual tools with language-model reasoning for image dialogue, visual mathematics, planning, and text reasoning.
- 5.2.3 Integrating Modalities Beyond the Visual Domain: Chameleon coordinates vision, web, knowledge, mathematics, and table tools through an LLM-driven planner for multistep tasks.
6 Multimodal Reasoning Benchmark Result Analysis
The benchmark analysis shows that MLLM reasoning performance varies substantially across evaluation settings, model families, and training strategies. Proprietary models generally lead broad benchmarks, while instruction tuning and language-model adaptation emerge as important factors, though the findings are case-study-specific.
- Benchmark results: GPT-4V achieves the best score on MM-Vet and leads InfiMM-Eval, while CogVLM-Chat is the strongest open-source model on MM-Vet.MM-Vet evaluates comprehensive open-ended capabilities, whereas InfiMM-Eval focuses on visual reasoning.
- Model factors: MLLM reasoning performance depends strongly on the underlying language model, with model strengths differing across reasoning complexity levels.Qwen-VL-Chat performs better on highly complex questions, whereas CogVLM-Chat is stronger on moderately complex questions.
- Benchmark results: Proprietary MLLMs outperform most open-source models on MMMU, while models retaining language-model capabilities perform comparatively well.The analysis suggests that retaining knowledge from LLM pretraining is important for college-level subject questions.
- Training strategies: Instruction tuning significantly improves performance on InfiMM-Eval, and unfreezing or adapting the language model is a recurring strategy among top-performing open-source models.SPHINX-v2, Qwen-VL-Chat, and CogVLM use language-model unfreezing or related adaptation during training.
- Cross-benchmark variation: Rankings differ substantially between InfiMM-Eval and MMMU, making it difficult to identify a single cause because datasets, language models, and training methods vary.The analysis warns that performance gaps across evaluation datasets should be expected when interpreting MLLM reasoning results.
- Scope and limitations: The conclusions are based mainly on top-performing models and InfiMM-Eval, so they may not generalize across training data, language models, or model scales.The authors explicitly characterize the analysis as a case study rather than a universal account.
7 Conclusions and Future Directions
The survey reviews MLLM reasoning and identifies architectural, training, context, data, reinforcement-learning, and evaluation limitations that shape future research directions.
- Conclusions: The survey synthesizes reasoning types, models, data, and evaluation methods to clarify the current state of MLLM reasoning.It also aims to inspire further exploration of reasoning abilities in future work.
- MLLM Architectures: Fixed image resolutions and perception-related hallucinations remain architectural obstacles to human-level multimodal reasoning.Additional modalities make it difficult to determine whether hallucinations arise from incomplete visual perception or language-model biases.
- Efficiency and Scalability of MLLM Training Recipes: Training recipes use vastly different data volumes without proportional performance differences, making data–model-scale relationships important to understand.Recent studies suggest vision-text alignment pretraining may not require extensive computational resources.
- Long-context Support: Long-context multimodal reasoning remains underexplored because MLLMs struggle with real-world long contexts and suitable datasets and benchmarks are lacking.The challenge includes controlling computational complexity and maintaining focus as prompt length increases.
- Instruction Fine-tuning (IFT) Data: Existing instruction-tuning datasets are insufficient for multimodal reasoning, while ablation evidence about effective data types remains limited.Future instruction datasets therefore require more careful design to improve reasoning capabilities.
- Reinforcement Learning for Multimodal Reasoning: Multimodal reinforcement learning still faces open problems in scalable preference-data collection, algorithm suitability, and tool-using agents.These issues arise from the complexity of tasks involving multiple modalities.
- Evaluation Benchmarks: Current benchmarks emphasize single-round conversations, leaving multi-round, multi-image evaluation underdeveloped for testing generalization.Such benchmarks could also help reduce over-fitting to specific instructions.