Source-linked AI summary
A Survey of Vision-Language Pre-Trained Models
Yifan Du, Zikang Liu, Junyi Li, Wayne Xin Zhao
TL;DR
VL-PTM research addresses how pre-training can improve vision-language learning and downstream task performance. This survey synthesizes modality encoding, interaction architectures, pre-training tasks, downstream tasks, and research directions, concluding that VL-PTMs have advanced beyond traditional methods while retaining substantial open challenges.
Problem
Vision-language learning needs pre-training approaches that effectively adapt representations across image-text modalities and improve downstream task performance, while comprehensive VL-PTM coverage has been lacking.
Method
The survey reviews image and text encoding, vision-language interaction architectures, pre-training tasks, downstream tasks, and promising research directions.
Results
VL-PTMs have made significant progress on vision-language tasks compared with traditional methods, with architecture suitability varying between reasoning and retrieval.
Takeaways & Limitations
VL-PTM research provides a synthesized account of current methods and pointers to related research while identifying directions for continued study.
Takeaways & Limitations
VL-PTMs remain difficult to deploy because their large models are computationally heavy, and several pre-training, compression, acceleration, and scaling challenges remain open.
Abstract
from arXiv · showhide
As transformer evolves, pre-trained models have advanced at a breakneck pace in recent years. They have dominated the mainstream techniques in natural language processing (NLP) and computer vision (CV). How to adapt pre-training to the field of Vision-and-Language (V-L) learning and improve downstream task performance becomes a focus of multimodal learning. In this paper, we review the recent progress in Vision-Language Pre-Trained Models (VL-PTMs). As the core content, we first briefly introduce several ways to encode raw images and texts to single-modal embeddings before pre-training. Then, we dive into the mainstream architectures of VL-PTMs in modeling the interaction between text and image representations. We further present widely-used pre-training tasks, and then we introduce some common downstream tasks. We finally conclude this paper and present some promising research directions. Our survey aims to provide researchers with synthesis and pointer to related research.
1 Introduction
Vision-language tasks jointly process images and text, but task-specific deep models transfer poorly. This survey reviews VL-PTMs, their architectures, pre-training tasks, downstream applications, and research directions.
- Vision-language tasks require jointly processing visual and linguistic information to answer questions or generate image descriptions.
- Task-specific deep models often transfer poorly, motivating pre-training on large-scale general datasets followed by downstream fine-tuning.
- VL-PTMs pre-train on large-scale image-text corpora to learn universal cross-modal representations for downstream V-L tasks.
- The survey addresses a gap by comprehensively reviewing VL-PTMs rather than primarily covering V-L tasks, datasets, traditional methods, or video-language models.
- The survey organizes VL-PTM pre-training around encoding modalities, modeling their interaction, designing pre-training tasks, and adapting learned features to downstream tasks.
2 Learning Vision-Language Representation
VL-PTM pre-training first converts text and images into semantic sequences of embeddings. Text commonly follows BERT-style encoding, while visual representation remains an open design problem involving regions, grids, patches, or transformers.
- VL-PTM pre-training begins by encoding images and texts into embeddings that preserve their semantics.
- The pre-training dataset consists of N paired texts and images, with text tokenized into tokens and images represented by object, grid, or patch features.
- BERT-style text representation combines token, positional, and segment embeddings after adding [CLS] and [SEP] markers.
- Visual representation must capture relationships among visual concepts, which are important for tasks such as image description generation.
- Visual encoders may use pixel-level grid features or transformer-based extraction, with some methods viewing the image globally to avoid neglecting critical regions.
3 Modeling Vision-Language Interaction
VL-PTMs model vision-language interaction with fusion or dual encoders. Fusion encoders support cross-modal reasoning but are computationally heavy, whereas dual encoders improve retrieval efficiency through separate modality encoding.
- Fusion Encoder: Fusion encoders integrate text and image inputs through attention and produce hidden states as fused multimodal representations.
- Single-stream Architecture: Single-stream fusion concatenates text and image embeddings with positional and modality indicators for unified transformer processing across task input formats.
- Dual-stream Architecture: Dual-stream architectures use bidirectional cross-attention between modalities while separately modeling intra-modal interactions.
- Architecture Trade-offs: Fusion encoders perform strongly on cross-modal reasoning but jointly encode all image-text pairs for matching, causing slow inference with heavy transformers.
- Dual Encoder: Dual encoders separately encode each modality, project embeddings into a shared semantic space, and efficiently compute similarity using shallow attention or dot products.
- Architecture Trade-offs: Architecture selection follows task demands: fusion encoders suit cross-modal reasoning, while dual encoders suit retrieval because they are more efficient.
- Hybrid Architectures: Hybrid models such as FLAVA and VLMo combine single-modal and cross-modal encoding to pursue benefits from both architectures.
4 Cross-Modal Pre-training Tasks
Cross-modal pre-training tasks shape what VL-PTMs learn by aligning masked language, masked visual regions, and image-text pairs at different levels. The survey also covers contrastive learning, which places matched and non-matched image-text representations in a shared semantic space.
- 4.1 Cross-Modal Masked Language Modeling: Cross-modal MLM predicts masked tokens using both unmasked text and vision features to align words with corresponding image content.Its masking strategy must prevent prediction from relying only on surrounding-token co-occurrence; image-dependent tokens encourage visual grounding.
- 4.2 Cross-Modal Masked Region Prediction: Cross-modal MRP masks visual regions and predicts them from remaining image features and text, learning object relationships and vision-language alignments.It includes MRC, which predicts masked-region semantic classes, and MRFR, which reconstructs masked high-dimensional region features.
- 4.2 Cross-Modal Masked Region Prediction: MRC uses cross-modal hidden states to predict masked-region labels, but detector-derived labels may be pseudo-labels whose quality depends on the object detector.Variants use soft detector outputs, while SOHO predicts masked visual tokens from surrounding tokens.
- 4.2 Cross-Modal Masked Region Prediction: MRFR reconstructs each masked region feature from its corresponding original feature, but random masking becomes less effective for grid or patch features because duplication can leak information.The survey notes that masking all features sharing a visual index can avoid this information leakage.
- 4.3 Image-Text Matching: ITM determines whether an image and text match by producing an alignment probability, providing coarse-grained cross-modal alignment alongside finer-grained MLM and MRP objectives.Models differ in how they form the single fused representation used to predict the alignment probability.
- 4.4 Cross-Modal Contrastive Learning: CMCL learns a shared vision-language semantic space by pulling matched image-text embeddings together and pushing non-matched pairs apart.Its contrastive loss is symmetrical, with text-to-image learning formulated similarly; CLIP and ALIGN exhibit zero-shot transfer to image classification.
5 Adapting VL-PTMs to Vision-Language Downstream Tasks
VL-PTMs support downstream vision-language tasks through cross-modal matching, reasoning, and generation. Architecture choice matters: fusion encoders favor reasoning, while dual encoders offer efficient retrieval.
- Downstream tasks are categorized into cross-modal matching, cross-modal reasoning, and vision-and-language generation.
- Cross-modal matching: Image-text retrieval matches sentences to images or images to sentences, with dual encoders enabling precomputed embeddings for efficient retrieval.
- Cross-modal matching: Visual referring expression localizes the image region corresponding to a textual description using region representations and matching scores.
- Cross-modal reasoning: VQA, NLVR, and VCR require reasoning over visual and textual information, with VQA commonly formulated as answer classification.
- Vision and language generation: Text-to-image generation produces images from descriptions, while multimodal text generation produces text conditioned on images and possibly other text.
- Fusion encoders are more suitable for cross-modal reasoning, whereas dual encoders retain similar retrieval performance with greater efficiency.
6 Conclusion and Future Directions
The survey reviews VL-PTM architectures, pre-training approaches, and downstream adaptation, while identifying unresolved directions. These include unified models, compression and acceleration, advanced pre-training, and scaling data and model capacity.
- The survey overviews VL-PTM architectures, pre-training approaches, and adaptation to downstream tasks, while identifying challenges for future research.
- Unified Model Architecture: Unified architectures aim to handle multiple modalities with a single transformer, but some attempts perform worse than task-specific baselines.
- Model Compression and Acceleration: VL-PTM deployment remains difficult because model size hinders real-life use, while quantization, pruning, and inference-speed improvements remain underexplored.
- Advanced Pre-training Methods: Adversarial samples and stage-wise pre-training show promise, but the potential of advanced pre-training methods is not fully developed.
- Reaching the Limit of VL-PTMs: ALIGN uses 675.4 million parameters and 1.8 billion image-text pairs, while future VL-PTMs are described as needing more high-quality data and parameters.