PaperlayerView exact revision on arXiv

Source-linked AI summary

VLP: A Survey on Vision-Language Pre-training

Feilong Chen, Duzhen Zhang, Minglun Han, Xiuyi Chen, Jing Shi, Shuang Xu, Bo Xu

arXiv:2202.09061v4cs.CVcs.CL

TL;DR

VLP asks whether uni-modal pre-training can support multimodal vision-language tasks amid limited high-quality labelled data. This survey synthesizes image-text and video-text VLP across five aspects, specific SOTA models, and emerging frontiers. It presents VLP as a rapidly advancing field while identifying efficiency and transfer challenges for future development.

  • Problem

    Multimodal tasks face limited high-quality labelled data, motivating the question of whether pre-training methods can be applied to vision-language tasks.

  • Method

    The paper surveys image-text and video-text VLP through feature extraction, architecture, objectives, datasets, downstream tasks, specific SOTA models, and new frontiers.

  • Results

    The survey provides a review of recent VLP advances and detailed summaries of specific SOTA VLP models across the field.

  • Takeaways & Limitations

    The survey aims to help researchers understand VLP and inspire new works that advance the field.

  • Takeaways & Limitations

    VLP model efficiency remains constrained: pruning and quantization remain underexplored, and only a few efforts improve model efficiency.

Abstract

from arXiv · show

In the past few years, the emergence of pre-training models has brought uni-modal fields such as computer vision (CV) and natural language processing (NLP) to a new era. Substantial works have shown they are beneficial for downstream uni-modal tasks and avoid training a new model from scratch. So can such pre-trained models be applied to multi-modal tasks? Researchers have explored this problem and made significant progress. This paper surveys recent advances and new frontiers in vision-language pre-training (VLP), including image-text and video-text pre-training. To give readers a better overall grasp of VLP, we first review its recent advances from five aspects: feature extraction, model architecture, pre-training objectives, pre-training datasets, and downstream tasks. Then, we summarize the specific VLP models in detail. Finally, we discuss the new frontiers in VLP. To the best of our knowledge, this is the first survey focused on VLP. We hope that this survey can shed light on future research in the VLP field.

1 Introduction

VLP applies pre-training to multimodal vision-language tasks, where models learn semantic correspondence between visual and textual modalities. This survey organizes the field’s advances, reviews SOTA models, and discusses new frontiers.

  • Motivation: VLP addresses limited high-quality labelled data in multimodal tasks by pre-training on large-scale data to learn vision-language semantic correspondence.Image-text models associate textual concepts with their visual appearance, while video-text models map textual objects and actions to video content.
  • Survey scope: The survey reviews VLP advances through feature extraction, model architecture, pre-training objectives, pre-training datasets, and downstream tasks.It discusses the basic details and goals of multimodal downstream tasks.
  • Survey scope: The survey summarizes specific SOTA VLP models in detail and discusses new frontiers in VLP.These discussions follow the reviews of core VLP components and downstream tasks.
  • Novelty: The authors describe this work as the first survey focused on VLP and aim to help researchers understand the field and design better models.The claim is distinguished from existing surveys focused on pretrained language or vision models.

2 Feature Extraction

VLP models convert image, video, and text inputs into counterpart features using detector-, CNN-, or ViT-based visual representations and pretrained-language-model text representations. Feature choices involve trade-offs between performance, efficiency, and model capacity.

  • Overview: VLP feature extraction preprocesses and represents images, videos, and text to obtain counterpart features.The survey covers visual and textual feature representations used before cross-modal modeling.
  • Image features: Object detectors produce region features with visual geometry embeddings, but region extraction is time-consuming and frozen detectors can limit model capacity.Faster R-CNN commonly supplies 2048-d RoI features and bounding-box geometry.
  • Image features: CNN-based methods extract grid features either end-to-end or after discretizing them with a learned vision dictionary.The resulting grid features are fed into the cross-modal module.
  • Image features: ViT-based methods reshape images into flattened 2D patches, with N = HW/P^2 patches serving as the Transformer input sequence length.The image is encoded as patch embeddings together with a [CLS] token embedding.
  • Video features: Video models extract frame features with CNN-based or ViT-based methods, then project visual features into the same lower-dimensional space as token embeddings.CNN features may come from ResNet, SlowFast, or I3D pretrained on image or video datasets.
  • Text features: Text inputs are segmented into subwords and represented by summed word, position, and type embeddings following pretrained language models.Examples include BERT, RoBERTa, AlBERT, and XLNet.
  • Feature encoders: VLP models can use randomly initialized Transformer encoders or pretrained visual and textual Transformers to encode modality-specific features.Examples of pretrained encoders include ViT, DeiT, and BERT.

3 Model Architecture

VLP architectures differ in how they fuse visual and textual features and how they generate outputs. They span single-stream or dual-stream fusion and encoder-only or encoder-decoder designs.

  • Architecture taxonomy: The survey classifies VLP architectures by multimodal fusion and overall architectural design.The two perspectives are single-stream versus dual-stream and encoder-only versus encoder-decoder.
  • Single-stream Architecture: Single-stream models concatenate text and visual features and feed them into one Transformer block using merged attention.Sharing parameters across modalities makes this architecture more parameter-efficient.
  • Dual-stream Architecture: Dual-stream models send text and visual features to separate Transformer blocks without shared parameters.Cross-attention can enable cross-modal interaction, while omitting it can improve efficiency.
  • Overall architecture: Encoder-only models directly feed cross-modal representations into an output layer to generate final outputs.This design contrasts with architectures that insert a decoder before the output layer.
  • Overall architecture: Encoder-decoder models first pass cross-modal representations through a Transformer decoder and then into an output layer.The decoder is the additional generation stage relative to encoder-only models.

4 Pre-training Objectives

VLP pre-training objectives guide models toward universal vision-language representations through four categories: completion, matching, temporal, and particular objectives.

  • VLP objectives are organized into completion, matching, temporal, and particular types.The survey presents these categories as its framework for reviewing pre-training objectives.
  • Completion: Completion reconstructs masked elements from the unmasked remainder to model a modality.
  • Matching: Matching projects vision and language into a shared hidden space for universal representations.
  • Temporal: Temporal objectives learn representations by reordering disrupted input sequences.
  • Particular: Particular objectives include downstream-style tasks such as visual question answering and visual captioning.

4.3 Masked Vision Modeling

Masked vision modeling reconstructs masked image or video features from remaining visual features and all textual features, using regression or semantic classification variants.

  • Masked vision modeling masks image or video regions or patches and reconstructs their features using remaining vision and all text.Visual features are typically masked with probability 15% and replaced by zeros.
  • Masked Feature Regression: Feature regression maps predicted masked features to the original feature dimension and applies L2 regression.
  • Masked Feature Classification: Masked feature classification predicts object semantic classes through a fully connected layer and softmax distribution.
  • Masked Feature Classification: Hard-label classification treats a detected object category as ground truth and minimizes cross-entropy against that pseudo-label.This approach relies on the detected category as the supervision target.
  • Masked Feature Classification: Soft-label classification uses the detector’s raw object-category distribution and minimizes KL divergence between distributions.The soft-label variant avoids assuming a single detected category is ground truth.

4.4 Vision-Language Matching

Vision-language matching objectives align modalities by representing paired vision and language jointly, either classifying pair matches or contrasting all in-batch pairings.

  • Vision-language matching projects vision and language into a shared space to align the modalities.Single-stream models use [CLS], while dual-stream models concatenate [CLSV] and [CLST] representations.
  • Vision-Language Matching: In single-stream models, the [CLS] representation serves as the fused representation for both modalities.
  • Vision-Language Matching: In dual-stream models, concatenated [CLSV] and [CLST] representations serve as the fused multimodal representation.
  • Vision-Language Contrastive Learning: Vision-language contrastive learning identifies matched pairs among N × N possible vision-language pairings in a batch of N pairs.The batch contains N^2−N negative pairings.
  • Vision-Language Contrastive Learning: Contrastive learning computes vision-to-text and text-to-vision similarities and applies cross-entropy losses in both directions.Similarity is often implemented with dot products and normalized with a temperature coefficient.

4.6 Word-Region Alignment

Word-region alignment uses optimal transport to align visual regions with words, while the survey also describes temporal ordering and downstream-task objectives among related VLP methods.

  • Word-Region Alignment: Word-region alignment uses optimal transport to learn unsupervised alignments between visual regions or patches and words.The IPOT algorithm approximates the otherwise computationally intractable exact minimization.
  • Word-Region Alignment: The resulting optimal-transport distance serves as the word-region alignment loss for training VLP models.
  • Frame Order Modeling: Frame order modeling randomly disrupts video-frame order and classifies each frame’s actual position.
  • Particular Pre-training Objects: VLP models may use visual question answering and visual captioning objectives to adapt pre-training to downstream tasks.VQA can be treated as classification over predefined answers or as direct answer generation.
  • Scope Boundary: The survey omits several specific pre-training objectives because of space limitations.Examples include grounding referring expression, image-conditioned denoising autoencoding, text-conditioned image feature generation, object detection, and aligned Kaleido patch modeling.

5 Pre-training Datasets

VLP pre-training datasets span image-text and video-text categories, differing in scale, sources, scenarios, and processing procedures. The survey highlights their central importance for cross-modal representation learning.

  • Pre-training datasets are important for cross-modal representation learning, with quality and size sometimes outweighing training strategies and algorithms.The survey therefore provides statistics and descriptions of widely used VLP datasets.
  • VLP datasets are broadly divided into image-language and video-language categories, with most studies combining multiple public datasets.Dataset size and sources vary across research works.
  • Image-language pre-training mainly uses image-text pairs, including human-annotated Flickr and COCO data alongside web-crawled CC3M and CC12M data.Some datasets target specialized settings, such as building-scale RGB-D scenes or fashion descriptions.
  • Video-language datasets are harder and more time-consuming to collect and process, limiting their scale relative to image-language data.Common sources include Internet videos and television programs, often paired with subtitles or dialogues.
  • Video-text datasets use varied annotations and processing, including action classes, captions, subtitles, and video concept labels.These procedures provide different strengths of alignment between video clips and text.

6 Downstream Tasks

VLP downstream tasks require cooperative vision-language knowledge across understanding, retrieval, generation, grounding, navigation, and recognition. The survey describes task inputs, outputs, and evaluation goals across image- and video-based settings.

  • VQA predicts an answer to a question about an image or video, typically by selecting the most suitable answer from a choice pool.Accurate performance requires inferring logical entailments from the visual input and question.
  • GQA extends VQA with structured semantic representations and evaluates consistency, validity, plausibility, distribution, and grounding.These multiple metrics enable analysis beyond traditional single-metric accuracy.
  • Video-language inference determines whether a hypothesis is entailed or contradicted by a video clip with aligned subtitles, while visual entailment uses three labels.The visual entailment labels are Entailment, Neutral, and Contradiction.
  • VLR retrieves relevant text for a visual query or relevant visuals for a text query using cross-modal matching strategies.The task supports domain-specific searches, search engines, and context-based visual retrieval systems.
  • VC generates semantically and syntactically appropriate descriptions for images or videos, requiring scene, entity, interaction, and language understanding.NoCaps tests description of novel objects unseen in the training corpus.
  • Other downstream tasks include visual dialogue, multimodal translation, navigation, OCR, entailment, commonsense reasoning, grounding, sentiment analysis, detection, segmentation, and action understanding.Their outputs range from free-form answers and translations to region scores, labels, continuous sentiment intensity, and action localization.

7 SOTA VLP models

The survey summarizes mainstream image-text and video-text VLP models, including their feature extraction, architectures, datasets, and downstream-task coverage. VisualBERT and VideoBERT illustrate early BERT-based single-transformer approaches for image-text and video-text inputs.

  • VisualBERT uses Faster R-CNN visual features concatenated with textual embeddings and feeds them into a single BERT-initialized transformer.Many later image-text VLP models follow this feature-extraction and architecture pattern while modifying objectives and datasets.
  • Table 2 summarizes mainstream image-text VLP models, distinguishing generic from domain-specific models by the number of downstream tasks.The table also records feature extraction, pre-training, embedding, and dataset notation.
  • Table 3 summarizes mainstream video-text VLP models using analogous information about feature extraction, pre-training, embeddings, datasets, and downstream tasks.Its caption identifies self-constructed/self-collected and multi-task-learning dataset abbreviations.
  • The survey reports that mainstream VLP models achieve state-of-the-art performance on many downstream tasks.
  • VideoBERT extends BERT to process videos and text simultaneously by concatenating ConvNet/S3D video features with word embeddings.Its ConvNet and S3D feature extractors are frozen during training, so the approach is not end-to-end.

8 Conclusion and New Frontiers

The survey concludes that VLP has progressed substantially while leaving open challenges in multimodal scope, knowledge, transfer, efficiency, domain shift, and architecture. It identifies research directions including audio integration, prompt tuning, compression, out-of-domain pre-training, and alternative architectures.

  • The survey presents the first VLP survey, reviewing five core aspects, summarizing state-of-the-art models, and discussing new frontiers.The five aspects are feature extraction, model architecture, pre-training objectives, pre-training datasets, and downstream tasks.
  • Prior multimodal pre-training often emphasizes language and vision while ignoring audio, leaving joint text-vision-audio representation learning open.Audio can provide emotion and acoustic-boundary information and support downstream tasks with acoustic inputs.
  • Existing VLP models primarily fit large-scale multimodal datasets, motivating unified cognitive architectures and knowledge-guided objectives for external common-sense and situational knowledge.
  • Fine-tuning is dominant but becomes parameter-inefficient as models grow and creates a gap between pre-training and diverse downstream tasks.Prompt tuning is proposed as a direction to reduce fine-tuning cost and bridge these stages.
  • VLP efficiency remains underexplored: knowledge distillation has been applied, while pruning and quantization remain open directions.Compression targets faster inference and deployment on resource-constrained devices.
  • In-domain pre-training data contributes to reported success, motivating out-of-domain pre-training for transfer to unknown data distributions.DeVLBert applies intervention-based learning inspired by backdoor adjustment to address pre-training and fine-tuning distribution bias.
  • Transformer-based VLP architectures may not be optimal, and diffusion models are identified as a possible alternative for future investigation.
Loading 2202.09061v4…