Source-linked AI summary
Multimodal Large Language Models: A Survey
Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, Philip S. Yu
TL;DR
Text-only language models struggle with non-text data, motivating multimodal models that combine diverse modalities. This survey organizes the field’s definitions, history, technical design, resources, applications, and challenges, concluding that multimodal models support more comprehensive and intelligent systems.
Problem
Pure-text large language models excel on text but have limited ability to understand and process other data types.
Method
The paper surveys multimodal concepts, historical algorithms, technical design choices, products, datasets, applications, and development challenges.
Results
The survey concludes that multimodal models enable machines to understand and process diverse data types and can support more comprehensive and intelligent systems.
Takeaways & Limitations
The paper provides researchers with technical guidance and datasets for experimentation and evaluation across multimodal models.
Takeaways & Limitations
Multimodal development still faces catastrophic forgetting, in which adapting language-trained networks to other tasks can reduce their original language capabilities.
Abstract
from arXiv · showhide
The exploration of multimodal language models integrates multiple data types, such as images, text, language, audio, and other heterogeneity. While the latest large language models excel in text-based tasks, they often struggle to understand and process other data types. Multimodal models address this limitation by combining various modalities, enabling a more comprehensive understanding of diverse data. This paper begins by defining the concept of multimodal and examining the historical development of multimodal algorithms. Furthermore, we introduce a range of multimodal products, focusing on the efforts of major technology companies. A practical guide is provided, offering insights into the technical aspects of multimodal models. Moreover, we present a compilation of the latest algorithms and commonly used datasets, providing researchers with valuable resources for experimentation and evaluation. Lastly, we explore the applications of multimodal models and discuss the challenges associated with their development. By addressing these aspects, this paper aims to facilitate a deeper understanding of multimodal models and their potential in various domains.
I. INTRODUCTION
Multimodal models combine diverse data types to address the perceptual limits of text-only language models. The paper surveys their definition, historical development, technical resources, applications, and challenges.
- Motivation: Multimodal models combine images, text, audio, and other data types, whereas pure-text LLMs have limited ability to process non-text inputs.The paper identifies multimodal perception as important for knowledge acquisition and interaction with the real world.
- Scope: The paper addresses prior reviews’ limited coverage of multimodal development and practical applications by organizing the field from foundational definitions through challenges.Its stated scope includes historical development and potential applications.
- Contributions: The survey defines multimodality, reviews historical algorithms, and provides practical guidance on knowledge representation, learning objectives, model construction, information fusion, and prompts.These contributions are presented as a technical guide for multimodal models.
- Resources: It compiles current algorithms and commonly used datasets as resources for future experimentation and evaluation.The dataset and algorithm review is presented as a basic research resource.
- Applications and challenges: The survey explores applications and development challenges while reporting that multimodal LLMs have shown superior common-sense reasoning to single-modality models.The reported performance observation is linked to cross-modal transfer for knowledge acquisition.
- Historical development: The historical account spans single-modality methods, modality conversion, modality fusion, and large-scale multimodal research.The stages cover developments from 1980 through the large-scale-model period beginning in 2020.
III. PRACTICAL GUIDE FOR TECHNICAL POINTS
The practical guide organizes multimodal model design around representation, objectives, architecture, fusion, and prompting. It explains how these choices support different multimodal tasks and training settings.
- Technical framework: Multimodal model design includes knowledge representation, learning-objective selection, model construction, information fusion, and prompt usage.These technical points are presented as the guide’s organizing framework.
- Knowledge representation: Text and images require tokenization and embedding, with image tokenization commonly using region-, grid-, or patch-based approaches.The guide contrasts text tokenization methods with the greater complexity of image tokenization.
- Learning objectives: Common multimodal pretraining objectives include image-text contrast, masked language modeling, masked visual modeling, and image-text matching.Image-text contrast aligns positive and negative image-text pairs through contrastive learning.
- Model construction: Encoder-only models directly process multimodal inputs and suit retrieval, while encoder-decoder models support generated outputs such as captions.The guide uses CLIP and ALBEF as encoder-only examples.
- Information fusion: Fusion encoders interact across modalities through self-attention or cross-attention, using single-stream or dual-stream approaches.Fusion is applied after modalities are encoded separately.
- Prompt usage: Prompting reduces the gap between pretraining and downstream fine-tuning, lowering fine-tuning costs and supporting zero- or small-data settings.The guide cites visual ChatGPT and CLIP as examples of informative prompt use.
IV. PRACTICAL GUIDE FOR ALGORITHMS
Multimodal algorithms are organized into foundation models and large-scale multimodal pre-trained models. The foundation model provides the basic framework for later large-scale multimodal pre-training.
- Multimodal algorithms comprise foundation models and large-scale multimodal pre-trained models.
- Foundation models serve as the basic framework for multimodal learning.
- Many novel large-scale multimodal pre-trained models are improved from foundation models.
A. Foundation model.
Foundation-model development adapts Transformer-based pre-training to visual data. ViT transfers Transformer mechanisms into vision, while BEiT adapts BERT-style generative pre-training for images.
- Transformer uses self-supervised pre-training on large-scale corpora followed by fine-tuning on downstream tasks.
- VIT: Vision Transformer applies Transformer mechanisms to computer vision while requiring careful consideration of input strategies.
- BEiT: BEiT adapts BERT to computer vision through generative pre-training.
- BEiT: BEiT discretizes image information into visual tokens and incorporates it into pre-training.
B. Large-scale multimodal pre-trained models
Large-scale multimodal models connect language models with visual encoders, foundation models, adapters, and instruction-tuning data. The surveyed approaches differ in how they transfer visual information and allocate trainable components.
- Visual ChatGPT combines visual foundation models with a Prompt Manager for iterative multi-step visual interactions.
- MM-REACT lets ChatGPT selectively use visual models as tools for multimodal tasks.
- Frozen concatenates visual-encoder embeddings with text and trains an autoregressive language model while keeping the LLM frozen.
- BLIP-2 uses a Qformer for image features and cross-attention, initially fine-tuning only the Qformer while freezing the visual encoder and LLMs.
- LLaMA-Adapter inserts trainable adapter prompt vectors into Transformer layers and adds projected visual features to the language model.
- MiniGPT-4 combines BLIP-2 and Vicuna, freezes their major components, fine-tunes a 15 M visual-side layer, and uses two-stage instruction data training.
- LLaVA generates diverse instruction data with GPT-4 and trains first on 600,000 image-text pairs before further instruction fine-tuning.
- PICa, PNP-VQA, and Img2LLM transfer image information into textual descriptions or question-answer demonstrations for LLM-based VQA.
V. PRACTICAL GUIDE FOR VARIOUS TASKS
Multimodal tasks translate, align, or fuse information across visual, textual, audio, and other modalities. The practical guide covers applications, representative model resources, and datasets for experimentation and evaluation.
- Image captioning: Image captioning converts visual representations into textual descriptions by modeling objects, actions, features, and relationships.
- Text-to-Image generation: Text-to-image generation translates textual prompts into novel images, with DALL-E 2 and Imagen presented as representative models.
- Sign language recognition: Sign language recognition converts camera-captured gestures into text by aligning temporal information between video and audio.
- Emotion recognition: Multimodal emotion recognition combines inputs such as video, text, audio, or sensor data, with late fusion suitable for music emotion recognition.
- Video processing: Video processing extends image-text multimodal models toward video-text and audio-text domains, producing models such as VideoCLIP and VideoCoCa.
- The guide categorizes representative multimodal datasets combining images, text, videos, and audio for experiments testing model effectiveness.
VI. CHALLENGES
The paper identifies modality expansion, computational demands, continual learning, and AGI-related issues as major challenges for multimodal models. These challenges span richer data integration, scalable training, persistent knowledge, and avoiding capability loss across tasks.
- Modalities expansion: Modalities must expand across diverse sensors and data sources to support more comprehensive analysis and recognition.The paper illustrates this with emotion computation using audio, facial expressions, ECG, and EEG.
- Modalities expansion: Combining CT, MRI, and PET data can provide complementary medical information for diagnosis and treatment decisions.CT describes tissue structure and lesions, MRI captures anatomy and tissue functionality, and PET detects metabolism and biomarkers.
- Time-consuming problem: Multimodal large models face time-consuming training because enormous computations, distributed clusters, multi-tenancy, fault tolerance, and multiple backbones must be supported.The paper frames these requirements as architectural and operational challenges for optimizing training and deployment.
- Lifelong/continual learning: Isolated learning leaves models without memory of learned knowledge, motivating lifelong or continual learning for real-world multimodal applications.The paper calls for models that learn continuously from experience and progressively improve their understanding of the world.
- Towards AGI: Catastrophic forgetting can reduce a model’s original language abilities when it is repurposed for other tasks, including robotic applications.The paper identifies smaller networks retrained from scratch and larger language backbones as two approaches discussed for mitigating this issue.
VII. CONCLUSION
The paper surveys multimodal model development from foundational concepts and historical algorithms through products, technical guidance, datasets, applications, and challenges. It aims to deepen understanding of multimodal models and their potential across domains.
- Scope: The paper defines multimodal models, traces algorithmic development, reviews technology-company products, and explains technical aspects.Its technical coverage includes practical guidance for understanding multimodal models.
- Resources: It compiles commonly used datasets as resources for experimentation and evaluation.The dataset compilation is presented alongside the survey’s algorithmic and technical coverage.
- Applications and challenges: The paper discusses multimodal applications and highlights development challenges for further research.The conclusion presents these discussions as part of its comprehensive exploration of multimodal model development.