PaperlayerView exact revision on arXiv

Source-linked AI summary

Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey

Xiao Wang, Guangyao Chen, Guangwu Qian, Pengcheng Gao, Xiao-Yong Wei, Yaowei Wang, Yonghong Tian, Wen Gao

arXiv:2302.10035v3cs.CVcs.AIcs.MM

TL;DR

Single-modality models leave broader multi-modal information challenges unresolved, motivating a comprehensive synthesis of large-scale multi-modal pre-trained models. The survey organizes their data, objectives, architectures, knowledge use, evaluations, and research directions, reporting close COCO captioning performance among representative models and identifying unresolved architectural and incremental-learning issues.

  • Problem

    Single-modality models leave information gaps, motivating research that incorporates multiple modalities; collecting and cleaning large-scale multi-modal data remains difficult.

  • Method

    The paper comprehensively reviews MM-PTMs, covering their background, task definitions, challenges, data, objectives, architectures, knowledge enhancement, downstream tasks, parameters, hardware, and results.

  • Results

    Representative models achieve close performance on COCO image captioning, with OSCAR obtaining 41.7, 30.6, 140, 24.5 and SimVLM achieving 40.6, 33.7, 143.3, 25.4, respectively.

  • Takeaways & Limitations

    The survey provides a broad basis for understanding cutting-edge MM-PTMs and proposes research directions for follow-up work.

  • Takeaways & Limitations

    Whether fusion or cross-modality matching modules designed for small-scale tasks work for large-scale pre-trained models remains to be verified.

Abstract

from arXiv · show

With the urgent demand for generalized deep models, many pre-trained big models are proposed, such as BERT, ViT, GPT, etc. Inspired by the success of these models in single domains (like computer vision and natural language processing), the multi-modal pre-trained big models have also drawn more and more attention in recent years. In this work, we give a comprehensive survey of these models and hope this paper could provide new insights and helps fresh researchers to track the most cutting-edge works. Specifically, we firstly introduce the background of multi-modal pre-training by reviewing the conventional deep learning, pre-training works in natural language process, computer vision, and speech. Then, we introduce the task definition, key challenges, and advantages of multi-modal pre-training models (MM-PTMs), and discuss the MM-PTMs with a focus on data, objectives, network architectures, and knowledge enhanced pre-training. After that, we introduce the downstream tasks used for the validation of large-scale MM-PTMs, including generative, classification, and regression tasks. We also give visualization and analysis of the model parameters and results on representative downstream tasks. Finally, we point out possible research directions for this topic that may benefit future works. In addition, we maintain a continuously updated paper list for large-scale pre-trained multi-modal big models: https://github.com/wangxiao5791509/MultiModal_BigModels_Survey. This paper has been published by the journal Machine Intelligence Research (MIR), https://link.springer.com/article/10.1007/s11633-022-1410-8, DOI: 10.1007/s11633-022-1410-8, vol. 20, no. 4, pp. 447-482, 2023.

1 Introduction

The survey motivates multi-modal pre-training as a response to the limitations of single-modality models and reviews the field’s development, scope, components, evaluations, and research directions.

  • Single-modality deep models have limited generalization, while larger datasets reduce this issue but require expensive and tedious collection and annotation.
  • Multi-modal models incorporate modalities such as RGB, depth, language, point clouds, audio, and event streams to address single-modality limitations.
  • The survey reviews multi-modal pre-training from its background through task definitions, challenges, benefits, data, architectures, objectives, and knowledge-enhanced methods.
  • It evaluates representative models through downstream tasks and reports model parameters, training hardware, and experimental results.
  • Compared with existing vision-language surveys, this review covers additional modalities, newer work from 2019 to June 2022, and categorized research directions.

2 Background

The background traces deep learning and large-scale pre-training across language, vision, and speech, emphasizing Transformer-based progress and the survey’s organization.

  • Deep learning models progressed from LeNet and AlexNet to VGG, ResNet, and DenseNet, with deeper architectures and improved performance supported by large-scale annotated data.
  • Large-scale pre-training exploits massive unlabeled data and offers advantages over small-scale methods in NLP, CV, and multi-modal fields while reducing annotation costs.
  • NLP pre-training advanced through self-supervised learning and Transformer structures, with BERT improving downstream tasks using fewer fine-tuning examples and GPT scaling parameters and data.
  • Vision pre-training adopted Transformer architectures, including sequence-based pixel prediction and ViT’s direct processing of image patches for classification.
  • Visual pre-training methods include masked image modeling, pixel regression, masked visual-token prediction, and perceptual similarity during visual codebook learning.
  • Speech pre-training uses contrastive learning, context prediction, masked speech modeling, and offline clustering to learn audio or speech representations.

3 Multi-Modal Pre-training

Multi-modal pre-training models are trained on massive, often weakly labeled data with large parameter scales and substantial computing requirements. The survey reviews their data, architectures, objectives, challenges, and practical advantages.

  • Task Definition: MM-PTMs use massive training datasets that are typically unlabeled because their scale makes annotation difficult.Their large parameter counts are tightly connected to multi-modal data and computing power.
  • Key Challenges: Collecting and cleaning multi-modal data is difficult because multi-modal devices are scarce and Internet data are noisy.Most current MM-PTMs focus on vision-language data because image and text are relatively accessible online.
  • Key Challenges: Architectures must tailor feature encoders to each modality and still require validation of large-scale fusion and cross-modality matching modules.The survey gives Transformer or CNN examples for image and text, and spiking networks for event streams.
  • Key Challenges: Designing effective multi-modal pre-training objectives remains challenging because existing methods often borrow masked prediction from single modalities.The survey identifies multi-modal-specific objectives as necessary alongside adaptations of single-modality tasks.
  • Key Challenges: Training MM-PTMs requires supercomputing resources because both multi-modal datasets and model parameters are large.The subsequent training process also requires substantial power support.
  • Key Challenges: Large-scale pre-training techniques have less accumulated experience than small-scale pre-training, making parameter tuning an important challenge.The survey notes that training an effective large model requires techniques beyond simply scaling existing methods.
  • Advantages: MM-PTMs support collaborative generation, modal completion, and cross-domain retrieval, and can extract common features across modalities.The survey also reports that multi-modal data can compensate for defects of single-modality data and provide additional prior knowledge.
  • Advantages: Self-supervised or unsupervised MM-PTMs can significantly improve generalizability compared with small-scale multi-modal models.The survey attributes this advantage to prior knowledge contained in massive data that small annotated datasets may not capture.

3.3 Pre-training Data

Large-scale multi-modal pre-training relies on diverse datasets spanning image-text, video-text, visual question answering, reasoning, retrieval, and multilingual settings. These resources differ in scale, annotation quality, modality coverage, and availability.

  • Dataset Overview: The survey presents an overview of multi-modal datasets proposed for large-scale pre-training.Table 2 summarizes dataset language and availability information.
  • Established Benchmarks: Flickr30k, COCO, Visual Genome, VQA v2.0, and FashionGen provide captioning, object-relation, question-answering, and expert-described fashion data.Their annotations include five captions per image in Flickr30k and COCO, and 13M answers for VQA v2.0.
  • Image-Text Datasets: CC12M expands CC3M by relaxing cleaning filters, producing a four-times larger dataset with a slight loss of accuracy.The expansion addresses the need for larger data in multi-modal pre-training.
  • Image-Text Datasets: Image-text resources range from 1M-scale caption collections such as SBU Captions and CC3M to billion-scale datasets such as AltText with 1.8B image-text pairs.AltText uses minimal frequency-based filtering, producing noisy data while supporting strong downstream performance.
  • Video-Language Datasets: Video-language datasets cover question answering and narrated instructional content, including TVQA's 152,545 question-answer pairs and HT100M's 136M video clips.HT100M descriptions are automatically transcribed and therefore weakly paired with videos.
  • Specialized and Diverse Resources: Other resources target visual reasoning, multilingual data, hierarchical labels, and ethical filtering through GQA, WIT, JFT-300M, JFT-3B, and RedCaps.RedCaps filters problematic images according to privacy and harmful-stereotype concerns, while WIT contains multilingual image-text sets.
  • Web-Scale Datasets: Web-scale datasets include Wukong with 100M image-text pairs, LAION-400M with 400M pairs, and YFCC-100M with 100M media objects.Wukong uses 200K queries for visual-concept diversity, while LAION-400M is filtered using CLIP.
  • Structured Annotations: Several datasets add specialized structure, including continuous semantic similarity ratings in CxC, object-instance composition in Product1M, and dense vision-language grounding in Localized Narratives.Localized Narratives combines spoken descriptions, textual descriptions, and mouse traces across 849K images.

3.4 Pre-training Objectives

MM-PTMs use diverse objectives to learn language, visual, cross-modal, video, and generative capabilities from largely unlabelled multimodal data.

  • Cross-modal objectives: Contrastive loss constructs positive and negative samples to learn relationships between modalities.CLIP and ALIGN use contrastive learning, while VinVL uses a 3-way contrastive loss.
  • Language objectives: Masked language modeling predicts randomly masked words using surrounding words and corresponding image regions as references.PrefixLM combines bidirectional attention over a prefix with autoregressive factorization over remaining tokens.
  • Vision objectives: Masked object classification predicts categories for masked image regions, whereas masked object regression predicts masked visual features or regions.LXMERT uses both MOC and MOR; detector-predicted object labels commonly provide MOC targets.
  • Alignment objectives: Image-text matching creates negative image-sentence pairs and uses binary classification to predict whether each pair is aligned.The similarity score is optimized against the ground-truth matching label.
  • Additional objectives: Other objectives include bidirectional and sequence-to-sequence language modeling, word-region alignment, action prediction, denoising, attribute and relation prediction, and image-text generation.Word-region alignment uses optimal transport, while image-text generation trains text generation conditioned on aligned visual input.
  • Video objectives: Video-text pre-training includes local and global subtitle-video alignment and frame-order modeling.Frame-order modeling treats reconstruction of selected video timestamps as a classification problem.

3.5 Pre-training Network Architecture

MM-PTM architectures encode modalities independently or jointly, then use attention, concatenation, or contrastive learning to model multimodal interactions.

  • Architectural overview: Transformer-based MM-PTMs commonly extract modality-specific embeddings before allowing modalities to interact.The survey distinguishes single-stream and cross-stream architectures by their information-fusion strategy.
  • Single-stream: Single-stream models tokenize and concatenate modality features in one transformer, using multi-head self-attention for interactive fusion.The fused representation is typically taken from the transformer’s class token.
  • Cross-stream: Cross-stream models encode modalities in parallel and align their representations through self-supervised contrastive learning.Their modality features can later be concatenated and passed to an MLP for objective learning.
  • Decoder design: Generative pre-training objectives require a decoder after the encoder to convert multimodal fusion features into outputs.Examples include masked image reconstruction and generating images from text descriptions.
  • Interaction modules: Attention-based interaction modules include merge-attention, co-attention, cross-attention, and tangled-transformer designs.These modules respectively share fusion attention, combine modality-specific self-attention with cross-attention, mutually query modalities, or inject modalities across transformer modules.
  • Interaction modules: Inter-modality contrastive learning models relationships by treating paired modality representations as positives and other pairings as negatives.CLIP and subsequent models use this approach for cross-modal relation modeling.

3.6 Pre-training using Knowledge

Knowledge-enhanced pre-training incorporates unstructured or structured knowledge to address logical reasoning and interpretability challenges, with dedicated knowledge evaluations.

  • Motivation: Knowledge-enhanced pre-trained models incorporate knowledge to address poor logical reasoning and limited interpretability in conventional pre-trained models.The survey calls these models Knowledge Enhanced Pre-Trained Models, or KEPTMs.
  • Knowledge representation: Knowledge representation learning mainly uses similarity-based models or graph neural networks to encode entities and relations.The survey reviews translation-based, tensor-factorization, neural-tensor, and graph-convolution approaches.
  • Taxonomy: KEPTMs are broadly divided into unstructured and structured knowledge models according to the form of supplied knowledge.The survey frames knowledge fusion after knowledge representation learning as an ongoing challenge.
  • Unstructured knowledge: Unstructured KEPTMs use plain text, words, phrases, entities, or linguistic information such as part-of-speech and sentiment tags.These approaches may encode knowledge implicitly in neural language models or support explicit retrieval during fine-tuning.
  • Structured knowledge: Structured KEPTMs incorporate syntax trees, rules, and knowledge graphs into pre-trained representations.Examples include syntax-aware Transformers and models trained across constituent and dependency parsing tasks.
  • Knowledge evaluation: Knowledge evaluation tasks test whether knowledge fusion improves semantic understanding beyond conventional performance metrics.The survey groups evaluations into factual and commonsense knowledge tasks, including question answering, fact verification, and commonsense generation.

3.7 Characteristics of Different Pre-trained Big Models

The survey contrasts early interactive-fusion MM-PTMs with newer large models, while also noting secondary development, unpaired-data training, and task-specific designs.

  • Model evolution: Early MM-PTMs such as ViLBERT and LXMERT use co-attention or cross-attention to interactively fuse multiple inputs.This design supports flexible integration with numerous downstream tasks.
  • Training and specialization: Some researchers extend existing large models, train with unpaired data to address shortages of paired multimodal data, or target specific domains and tasks.The survey cites e-commerce and indoor navigation as examples of specialized settings.

4 Downstream Tasks

The survey organizes downstream validation of MM-PTMs into generative, classification, and regression tasks, while also reviewing prompt learning for adapting tasks to pre-trained objectives.

  • Downstream validation covers generative, classification, and regression tasks, with representative prompt-learning algorithms also reviewed.
  • Generative Tasks: Image and video captioning encode visual inputs before predicting sentences word by word with a language decoder.
  • Generative Tasks: Visual question answering maps an image and question to an answer, while visual commonsense reasoning additionally requires a rationale.
  • Classification and Regression Tasks: Category recognition predicts image categories, whereas multi-modal sentiment analysis is described as a multi-modal fusion task.
  • Classification and Regression Tasks: Grounding referring expressions predicts the location of an object from an image and language description, with video variants extending grounding across space and time.
  • Prompt Learning: Prompt learning reformulates downstream tasks to match pre-training objectives, and prompt tuning can freeze the large model while adjusting prompts or task-specific parameters.

5 Experimental Analysis

The survey analyzes the scale, compute requirements, and downstream results of representative MM-PTMs. It finds rapidly increasing research activity, substantial hardware demands, varied retrieval performance, and close captioning results among compared models.

  • MM-PTM papers increased year by year from 2019 to 2022, with 2022 counting only January through June.
  • Model Parameters and Training Information: Many selected models require more than 100 GPUs for training, including SimVLM, MURAL, CLIP, VATT, and Florence.
  • Model Parameters and Training Information: Many MM-PTMs have limited parameter counts, while DALLE-E, BriVL, M6, and CogView reach 12000 MB, 10000 MB, 100000 MB, and 4000 MB, respectively.
  • Downstream Results: Zero-shot image-retrieval performance differs substantially across MM-PTMs, with Rank-1 and Rank-5 reported as separate measures.
  • Downstream Results: On COCO image captioning, OSCAR, VinVL, and SimVLM achieve close results across BLEU, METEOR, CIDEr, and SPICE, and outperform supervised ImageNet-pretrained models.

6 Research Directions

The survey identifies research directions spanning broader modalities, incremental and knowledge-enhanced pre-training, fine-grained representations, prompt learning, and cross-modal architecture design, motivated by data, cost, and flexibility challenges.

  • Pre-training on More Modalities: Large-scale MM-PTMs should incorporate more modalities because existing systems usually use vision and language, while aligned multimodal data remain limited.
  • Incremental Learning based Pretraining: Incremental learning could reduce redundant and costly retraining when new data arrive, but its suitability for big models remains unclear.
  • Incremental Learning based Pretraining: Modality-incremental learning should help models absorb new sensors appearing in the future while retaining flexibility.
  • Knowledge Enhanced Multi-Modal Pre-training: Knowledge-enhanced pre-training needs multimodal-specific knowledge, general fusion methods beyond vision and language, and dedicated pre-training knowledge evaluations.
  • Fine-grained Multi-Modal Pretraining: Fine-grained pre-training is proposed to complement global image-language matching by mining local or instance-level information.
  • Multi-Modal Pre-trained Model based Prompt Learning: Prompt learning addresses the gap between modality-matching or masked-token pre-training objectives and classification or regression downstream tasks.
  • Architecture Design: Future architectures should study both cross-modal coupling for correlation and decoupling for dynamically expanding modalities.

7 Conclusion

The paper surveys MM-PTMs from their background and core components through downstream tasks, model scale, hardware, and representative experiments, then identifies directions for future study.

  • The survey covers MM-PTM background, task definition, challenges, benefits, data, objectives, networks, and knowledge-enhanced pre-training.
  • It reviews generative, classification, and regression tasks, analyzes model parameters and pre-training hardware, and visualizes representative experimental results.
  • The paper concludes by identifying research directions intended to provide insights for future MM-PTM work.

Publish Information

The paper was published in Machine Intelligence Research.

  • The paper appeared in the journal Machine Intelligence Research.
Loading 2302.10035v3…