PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Multimodal Large Language Models

Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, Enhong Chen

arXiv:2306.13549v4cs.CVcs.AIcs.CLcs.LG

TL;DR

MLLM research needs a consolidated account of rapidly developing multimodal models that combine LLM reasoning with multimodal information. This survey synthesizes their foundations, evaluation, extensions, hallucination research, techniques, challenges, and future directions, concluding that the field still has substantial open research space.

  • Problem

    Text-only LLMs cannot understand vision, while large vision models commonly lag in reasoning, motivating a survey of the rapidly developing MLLM field.

  • Method

    The paper surveys MLLM architectures, training strategies and data, evaluation, extensions, multimodal hallucination, M-ICL, M-CoT, and LLM-Aided Visual Reasoning.

  • Results

    The survey provides a broad view of MLLM progress, including basic recipes, related extensions, current research gaps, and promising directions.

  • Takeaways & Limitations

    MLLMs exhibit strong generalization, emergent abilities, and improved interactivity and control compared with conventional visual reasoning models.

  • Takeaways & Limitations

    MLLMs still struggle with complicated instructions, and current M-ICL and M-CoT research remains rudimentary with weak related capabilities.

Abstract

from arXiv · show

Recently, Multimodal Large Language Model (MLLM) represented by GPT-4V has been a new rising research hotspot, which uses powerful Large Language Models (LLMs) as a brain to perform multimodal tasks. The surprising emergent capabilities of MLLM, such as writing stories based on images and OCR-free math reasoning, are rare in traditional multimodal methods, suggesting a potential path to artificial general intelligence. To this end, both academia and industry have endeavored to develop MLLMs that can compete with or even better than GPT-4V, pushing the limit of research at a surprising speed. In this paper, we aim to trace and summarize the recent progress of MLLMs. First of all, we present the basic formulation of MLLM and delineate its related concepts, including architecture, training strategy and data, as well as evaluation. Then, we introduce research topics about how MLLMs can be extended to support more granularity, modalities, languages, and scenarios. We continue with multimodal hallucination and extended techniques, including Multimodal ICL (M-ICL), Multimodal CoT (M-CoT), and LLM-Aided Visual Reasoning (LAVR). To conclude the paper, we discuss existing challenges and point out promising research directions. In light of the fact that the era of MLLM has only just begun, we will keep updating this survey and hope it can inspire more research. An associated GitHub link collecting the latest papers is available at https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models.

1 INTRODUCTION

MLLMs combine LLM-based reasoning with multimodal input and output, addressing the complementary limitations of text-only LLMs and vision models. This survey organizes the field’s architectures, training, evaluation, extensions, hallucination research, techniques, challenges, and directions.

  • Motivation: MLLMs combine LLM-based reasoning with multimodal information processing, emerging from the complementarity between text-only LLMs and vision models.LLMs are strong at reasoning but cannot directly understand vision, while large vision models perceive visual information but commonly lag in reasoning.
  • Distinctive traits: Unlike traditional multimodal models, MLLMs use billionscale LLMs and multimodal instruction tuning to support capabilities such as image-based code generation and OCR-free math reasoning.The survey identifies large-scale LLM backbones and new training paradigms as representative MLLM traits.
  • Research progress: Research has expanded from image-, video-, and audio-grounded text generation toward finer-grained controls, broader capabilities, and additional usage scenarios.Examples include specifying regions with boxes or objects with clicks.
  • Survey contribution: The authors position the work as a broad survey of MLLM progress and provide a GitHub page intended to track additional literature over time.The paper states that the associated resource is updated in real time and describes the survey as the first on MLLMs to the authors’ knowledge.
  • Survey scope: The survey reviews MLLM architectures, training strategies and data, evaluation practices, extensions, multimodal hallucination, M-ICL, M-CoT, and LLM-Aided Visual Reasoning.Its structure moves from foundational components and evaluation to extensions, hallucination mitigation, specialized techniques, and future directions.

2 ARCHITECTURE

MLLMs combine pretrained modality encoders, connectors, and LLMs, optionally adding generators for non-text outputs. Architectural choices span encoder scale and resolution, LLM capacity, fusion strategy, and modality interfaces.

  • Overall architecture: A typical MLLM uses a pretrained modality encoder, modality interface, and LLM, with an optional generator for outputs beyond text.Encoders process modality signals, interfaces align them with language, and LLMs perform understanding and reasoning.
  • Modality encoder: Pretrained encoders compress images or audio into representations and provide an aligned starting point for connecting modalities to LLMs.CLIP is cited as an image encoder aligned with text through large-scale image-text pretraining.
  • Scaling: Higher input resolution and larger LLMs generally improve MLLM performance, while larger models can exhibit emergent capabilities such as zero-shot Chinese understanding.Studies report gains from scaling LLMs from 7B to 13B and from using larger models across MLLM benchmarks.
  • Scaling: Mixture-of-Experts architectures scale total parameters through selective activation and empirically outperform dense counterparts on almost all reported benchmarks.The cited studies use sparse activation to increase parameter capacity without increasing computational cost.
  • Modality interface: Connectors bridge modality features and language through projection-based, query-based, or feature-level fusion, with token-level fusion commonly concatenating visual tokens with text.Query-based interfaces compress visual information into learnable query tokens, while feature-level methods add deep cross-modal interaction inside the LLM.
  • Modality interface: For token-level fusion, visual-token count and input resolution matter more than adapter type, while token-level fusion empirically outperforms feature-level fusion on VQA benchmarks.The reported performance gap may reflect the more complicated hyperparameter search required by cross-attention models.

3 TRAINING STRATEGY AND DATA

Full-fledged MLLMs are trained in three stages, with each stage using distinct data and serving a different objective.

  • Training stages: MLLM training consists of pre-training, instruction-tuning, and alignment tuning, each requiring different data types and fulfilling different objectives.The survey organizes the training discussion around objectives, data collection, and data characteristics for each stage.

3.1 Pre-training

Pre-training aligns modalities and learns multimodal world knowledge, typically by training on large-scale text-paired data such as image captions. The survey describes autoregressive caption prediction and increasingly fine-grained data construction.

  • Objective and data: Pre-training primarily aligns modalities and learns multimodal world knowledge from large-scale text-paired image, audio, or video data.Caption pairs describe visual or acoustic content in natural-language sentences.
  • Training detail: Given an image, the model autoregressively predicts its caption, commonly training a learnable interface while keeping pretrained encoders and LLMs frozen.Some methods unfreeze additional modules to increase the number of trainable parameters.
  • Data construction: Fine-grained data generated with strong MLLMs provides longer and more accurate descriptions but costs more and is available at smaller volume.ShareGPT4V balances quality and scale by training a captioner on 100K GPT-4V-generated examples before expanding to 1.2M data.
  • Survey organization: The survey compares pre-training datasets and learning paradigms as part of its review of training data and procedures.Figure 3 compares three typical learning paradigms, while Table 4 summarizes common pre-training datasets.

3.2 Instruction-tuning

Instruction tuning teaches MLLMs to follow natural-language task descriptions across multimodal inputs and outputs. The section covers sample structure, autoregressive training, scalable data collection, and the importance of data quality.

  • 3.2.1 Introduction: Instruction tuning teaches models to understand user task descriptions and generalize to unseen tasks, improving zero-shot performance.
  • 3.2.1 Introduction: Compared with task-specific fine-tuning and prompting, instruction tuning supports broader task following through varied instruction formulations.
  • 3.2.1 Introduction: The survey organizes this section around instruction-sample formats, training objectives, instruction-data collection, and commonly used datasets.
  • 3.2.2 Training Detail: A multimodal instruction sample combines an optional task description with multimodal input and a ground-truth response, including image-only captioning cases.
  • 3.2.2 Training Detail: MLLMs are typically trained with the autoregressive language-model objective, predicting response tokens conditioned on the instruction and multimodal input.
  • 3.2.3 Data Collection: Instruction data can be harvested by adapting existing task datasets, using manually designed or GPT-assisted templates, and generating samples through self-instruction.
  • 3.2.3 Data Collection: Language-only and multimodal conversations can be combined through mixed or sequential instruction tuning to improve conversational and instruction-following abilities.
  • 3.2.4 Data Quality: Instruction-data quality matters alongside quantity: diverse prompts, visual reasoning tasks, and more complex instructions can improve performance and generalization.

3.3 Alignment tuning

Alignment tuning adapts MLLMs to human preferences, especially preferences concerning response quality and hallucination reduction. The survey presents RLHF and DPO, their training mechanics, and multimodal preference-data applications.

  • 3.3 Alignment tuning: Alignment tuning targets scenarios requiring models to match specific human preferences, including responses with fewer hallucinations.
  • 3.3 Alignment tuning: The section introduces RLHF and DPO, explains their main ideas, and compiles datasets used for practical alignment problems.
  • 3.3 Alignment tuning: RLHF uses human annotations in a loop involving supervised fine-tuning, reward modeling from comparisons, and reinforcement learning with PPO.
  • 3.3 Alignment tuning: A per-token KL penalty helps keep the reinforcement-learning policy from deviating too far from the original policy.
  • 3.3 Alignment tuning: The RL policy and reference model are initialized from the supervised model, after which the RL policy is expected to align with human preferences.
  • 3.3 Alignment tuning: DPO learns directly from human preference labels with binary classification, avoiding an explicit reward model and simplifying the RLHF pipeline.
  • 3.3 Alignment tuning: Multimodal alignment methods collect preference data by correcting hallucinations at segment level or distilling GPT-4V preference supervision into instruction-tuned models.
  • 3.3 Alignment tuning: Alignment-feedback collection is generally more expensive and uses less data than earlier training stages; LLaVA-RLHF contains 10K human-feedback preference pairs.

4 EVALUATION

MLLM evaluation must cover versatile and emergent capabilities across both closed-set and open-set settings. The survey reviews benchmark metrics, comprehensive benchmarks, human and GPT-based scoring, and qualitative case studies, while noting limitations of text-only judges.

  • MLLM evaluation requires comprehensive coverage because models are versatile and exhibit emergent capabilities such as OCR-free math reasoning.
  • Closed-set evaluation: Closed-set evaluation uses predefined answer options, task-specific datasets, and benchmark metrics under zero-shot or finetuning settings.Examples include accuracy on ScienceQA and CIDEr on NoCaps and Flickr30K.
  • Closed-set evaluation: New MLLM benchmarks address the limited task and dataset coverage of earlier evaluation methods, including MME's 14 perception and cognition tasks.
  • Open-set evaluation: Open-set responses are harder to judge because their content is flexible, motivating manual scoring, GPT scoring, and case studies.
  • Open-set evaluation: Text-only GPT-4 judges image-related text without seeing the image, making its use as a performance upper bound questionable; GPT-4V can assess responses with direct image access.

5 EXTENSIONS

MLLM research extends model inputs, outputs, modalities, languages, and deployment scenarios. These extensions move interaction toward finer-grained visual control, mixed-modality processing, multilingual support, and resource-limited or real-world use.

  • Granularity Support: Granularity has progressed from whole-image interaction to region- and pixel-level inputs, including bounding boxes, points, and sketches.Shikra supports region-level understanding, while Ferret and Osprey support more flexible referring schemes.
  • Modality Support: Mixed-modality frameworks combine text, image, audio, and video by attaching diffusion models to an encoder-decoder MLLM with the LLM as a reasoning pivot.
  • Language Support: Language support remains limited because current models are predominantly unilingual and high-quality non-English training data is scarce.
  • Scenario Support: Scenario-specific MLLMs target resource-limited and real-world settings, using smaller models, quantization, and agents for interactions such as graphical-user-interface assistance.

6 MULTIMODAL HALLUCINATION

Multimodal hallucination occurs when MLLM responses conflict with image content, making its measurement and mitigation a central research problem. The survey organizes hallucination types, evaluation metrics, and correction methods across pre-, in-process, and post-correction.

  • Multimodal hallucination is the generation of responses inconsistent with image content and is treated as a fundamental research problem.
  • Hallucination Types: Hallucinations comprise existence errors, attribute errors, and more complex relationship errors involving objects and their interconnections.Attribute hallucination is typically associated with existence hallucination because attributes should be grounded in present objects.
  • Evaluation: CHAIR measures hallucinated objects in open-ended captions, whereas POPE evaluates object existence through binary closed-set questions and robustness settings.
  • Evaluation: Text-only judges require reference annotations and have limited image context, while Woodpecker uses GPT-4V to assess responses directly against the image.
  • Mitigation: Mitigation methods are grouped into pre-correction, in-process correction, and post-correction according to when hallucinations are addressed.
  • Mitigation: Pre-correction uses specialized data or preference optimization, while in-process methods control imagination or counter language priors and post-correction remedies generated outputs.

7 EXTENDED TECHNIQUES

The survey presents multimodal extensions of in-context learning and chain-of-thought reasoning, alongside systems that use LLMs and external tools for visual reasoning. These techniques support few-shot adaptation, structured reasoning, multimodal generation, and stronger generalization and interaction.

  • Multimodal ICL: Multimodal ICL adds demonstration samples to an input at inference time, preserving the flexible, training-free nature of learning from analogy.The number and ordering of examples can be adjusted, and Table 9 illustrates a structured query template.
  • Multimodal ICL: M-ICL research combines instruction tuning, extra output modalities, unified image-text representations, and context schemes for related images.
  • Multimodal ICL: M-ICL is mainly applied to visual reasoning and teaching LLMs to use external tools through task-specific examples and output templates.
  • Multimodal CoT: M-CoT extends chain-of-thought reasoning to multimodal inputs and can be acquired through finetuning, few-shot learning, or zero-shot learning.These approaches require decreasing sample sizes in that order.
  • Multimodal CoT: M-CoT chains vary in structure and construction, including single-chain or tree-shaped reasoning and infilling-based or predicting-based patterns.
  • LLM-Aided Visual Reasoning: LLM-aided visual reasoning systems invoke external tools or vision foundation models and exhibit strong generalization, emergent abilities, and improved interaction control.Examples include explaining why a meme is funny and supporting click or natural-language queries.

8 CHALLENGES AND FUTURE DIRECTIONS

The survey identifies unresolved challenges in multimodal context processing, instruction following, advanced reasoning techniques, embodied agents, and safety. It also highlights corresponding directions for extending MLLMs into more capable and reliable systems.

  • Current MLLMs struggle with long multimodal contexts, limiting long-video understanding and documents interleaved with images and text.
  • MLLMs need stronger instruction-following capabilities because models other than GPT-4V generally fail to generate high-quality question-answer data from complicated instructions.
  • Multimodal in-context learning and chain-of-thought techniques remain rudimentary, with weak related capabilities and substantial room for improvement.
  • Embodied MLLM agents require perception, reasoning, planning, and execution to interact with the real world.
  • MLLMs remain vulnerable to crafted attacks that can elicit biased or undesirable responses, making safety improvement an important research topic.

9 CONCLUSION

The paper surveys existing MLLM literature to present a broad view of its main directions, basic recipe, and extensions. It identifies research gaps and promising directions to clarify current progress and inspire further work.

  • The survey synthesizes MLLM literature across its main directions, including the basic recipe and related extensions.
  • It highlights current research gaps and points out promising directions for future investigation.
  • The authors aim to provide a clear picture of MLLM progress and inspire additional research.
Loading 2306.13549v4…