Source-linked AI summary
mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections
Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng Cao, Ji Zhang, Songfang Huang, Fei Huang, Jingren Zhou, Luo Si
TL;DR
Existing vision-language models face inefficient full self-attention over long visual sequences and information asymmetry between detailed images and short abstract captions. mPLUG uses an asymmetric cross-modal architecture with skip-connections and unimodal encoders for understanding and generation. It achieves state-of-the-art results across vision-language tasks and strong zero-shot transfer to video-language tasks.
Problem
Vision-language fusion is challenged by the high computation of full self-attention on long visual sequences and asymmetry between detailed images and short abstract captions.
Method
mPLUG combines unimodal image and text encoders with asymmetric co-attention, connected-attention and cross-modal skip-connections for vision-language understanding and generation.
Results
mPLUG achieves state-of-the-art performance across vision-language tasks and strong zero-shot transfer to multiple video-language tasks.
Takeaways & Limitations
The framework supports effective and efficient cross-modal learning across both understanding and generation settings.
Abstract
from arXiv · showhide
Large-scale pretrained foundation models have been an emerging paradigm for building artificial intelligence (AI) systems, which can be quickly adapted to a wide range of downstream tasks. This paper presents mPLUG, a new vision-language foundation model for both cross-modal understanding and generation. Most existing pre-trained models suffer from the problems of low computational efficiency and information asymmetry brought by the long visual sequence in cross-modal alignment. To address these problems, mPLUG introduces an effective and efficient vision-language architecture with novel cross-modal skip-connections, which creates inter-layer shortcuts that skip a certain number of layers for time-consuming full self-attention on the vision side. mPLUG is pre-trained end-to-end on large-scale image-text pairs with both discriminative and generative objectives. It achieves state-of-the-art results on a wide range of vision-language downstream tasks, such as image captioning, image-text retrieval, visual grounding and visual question answering. mPLUG also demonstrates strong zero-shot transferability when directly transferred to multiple video-language tasks.
1 Introduction
mPLUG addresses the efficiency and information-asymmetry challenges of cross-modal fusion with an asymmetric architecture and cross-modal skip-connections. It targets both effective understanding and generation while reporting broad task performance and zero-shot transfer.
- Long visual sequences make full self-attention computationally expensive, while short abstract captions create information asymmetry during multimodal fusion.
- Connected-attention networks fuse visual and linguistic features early but require full self-attention over concatenated sequences and process both modalities equally.
- Separate visual and textual Transformers alleviate information asymmetry but remain inefficient on long visual sequences and use two networks.
- mPLUG introduces asymmetric cross-modal skip-connections that fuse representations at disparate abstraction levels and skip visual layers.The shortcuts are intended to reflect the semantic richness of language relative to vision.
- The architecture combines early asymmetric co-attention with connected-attention, and its shortcuts alleviate vanishing gradients during training.The first few layers remove co-attention on the vision side before one connected-attention layer concatenates visual and language-side representations.
- mPLUG is a unified pretrained model for cross-modal understanding and generation, achieving state-of-the-art results across vision-language tasks and strong zero-shot transfer to video-language tasks.
2 Related Work
Vision-language pretraining commonly uses dual encoders for efficient retrieval or fusion encoders for fine-grained interaction. mPLUG extends this landscape with cross-modal skip-connections that target fusion efficiency and modality asymmetry.
- Dual encoders separately encode images and text, enabling cached representations and computation-efficient retrieval but struggling with complex reasoning tasks such as visual question answering.
- Fusion encoders use self-attention or cross-attention to model fine-grained interactions between image and text sequences.
- Pixel-BERT, E2E-VLP and ViLT remove object detectors to improve inference speed, while VLMo unifies dual- and fusion-encoder modules with shared modality experts.
- Skip-connections address optimization problems in deep networks by bypassing gradient exploding or vanishing, with designs including summed, gated and concatenated shortcuts.
- mPLUG proposes cross-modal skip-connections specifically to address cross-modal fusion challenges.
3 mPLUG
mPLUG combines separate image and text encoders with a cross-modal skip-connected network and decoder, using asymmetric fusion and multiple discriminative and generative pre-training objectives.
- 3.1 Model Architecture: mPLUG uses separate visual and text encoders, a cross-modal skip-connected network, and a Transformer decoder for sequence-to-sequence learning.
- 3.2 Cross-modal Skip-connected Network: Cross-modal fusion occurs through connected-attention layers interleaved with S asymmetric co-attention layers, repeated across N fusion blocks.The fixed stride S determines how often connected cross-modal fusion is applied.
- 3.2 Cross-modal Skip-connected Network: Asymmetric co-attention updates text using visual features, while connected attention jointly processes the connected image and text representation.The asymmetric block comprises self-attention, cross-attention, and a feed-forward network.
- Pre-training Objectives: mPLUG jointly optimizes image-text contrastive learning, image-text matching, masked language modeling, and prefix language modeling.The objectives cover three understanding tasks and one generation task.
4 Distributed Learning on a Large Scale
Large-scale mPLUG training addresses memory and computation bottlenecks through distributed parameter partitioning, checkpointing, and BF16 precision.
- Training throughput is increased by reducing memory usage and computation time.
- ZeRO partitions parameters, optimizer states, and gradients across data-parallel GPUs, reducing single-GPU static memory overhead to approximately 1/N.Here, N denotes the number of GPU cards.
- Gradient checkpointing reduces runtime memory by recomputing activation values during backpropagation instead of retaining them.This reduces memory at the cost of increased forward computation time.
- BF16 precision reduces computation time while matching FP32’s representation range and FP16’s fast computing speed.
5.1 Data & Setup
mPLUG is pretrained on 14M image-text examples for 30 epochs using large-scale distributed training, with Transformer encoders and decoder configurations described for base and large models.
- Data: The pre-training dataset contains 14M image-text examples from two in-domain and three web outdomain datasets.The in-domain datasets are MS COCO and Visual Genome; the web datasets include Conceptual Captions, Conceptual 12M, and SBU Captions.
- Setup: Training runs for 30 epochs with total batch size 1024 on 16 NVIDIA A100 GPUs.
- Setup: The text encoder and cross-modal skip-connected network use 6-layer Transformers, while the decoder uses a 12-layer Transformer.
- Setup: The visual encoder uses CLIP-pretrained ViT-B/16 for the base architecture and ViT-L/14 for the large architecture.
- Evaluation: Table 1 evaluates captioning on the COCO Caption Karpathy test split and NoCaps validation set, while Table 2 evaluates VQA on its test set.
5.2 Evaluation on Vision-Language Tasks
mPLUG is evaluated across image captioning, image-text retrieval, visual grounding, and visual reasoning tasks, achieving strong or state-of-the-art results across these benchmarks.
- Visual Question Answering: mPLUG treats VQA as unconstrained open-vocabulary answer generation rather than classification over predefined answer sets.
- Image Captioning: mPLUG outperforms state-of-the-art captioning models on COCO Caption and NoCaps using only 14M pretraining images.On COCO Caption, it leads the Karpathy test set by 5.5 CIDEr points.
- Image-Text Retrieval: +2.7% and +1.0% in COCO and Flickr30K TR recall@1, respectively, over BLIP with the same 14M pretraining images.
- Visual Grounding: 3.16% and 1.22% absolute gains over OFA are reported on RefCOCO testB and RefCOCOg test-u visual grounding, respectively.
- Visual Reasoning: mPLUG achieves competitive performance on NLVR2 and SNLI-VE, outperforming SimVLM and BLIP despite their larger pretraining datasets.
5.3 Effectiveness and Efficiency
The analysis shows that cross-modal skip-connections improve the efficiency–performance tradeoff, while training optimizations substantially increase throughput.
- Analysis of Stride for Skip: Performance increases through S = 3 and slightly decreases afterward, making stride the key efficiency–effectiveness control.
- Analysis of Stride for Skip: 5X lower running time is achieved when stride increases from S = 1 to S = 6 by skipping vision co-attention layers.
- Analysis of Stride for Skip: Nearly 30% faster running time is obtained at S = 6 with comparable performance to S = 3, motivating S = 6 for mPLUGViT-L pretraining.
- Cross-modal Fusion Variants: Co-attention and connected-attention require substantially more running time because of the long visual sequence.
- Training Throughput: Training throughput rises from 124 to 422 samples per second, a 3X improvement from memory-saving and accelerated-training techniques.
5.4 Zero-shot Transferability
mPLUG transfers directly from image-text pretraining to vision-language and video-language tasks without task-specific video pretraining or supervision, achieving competitive or superior zero-shot results.
- Zero-shot Vision-Language Tasks: Zero-shot image captioning is competitive with fully supervised Oscar and VinVL baselines.
- Zero-shot Vision-Language Tasks: Zero-shot Flickr30K retrieval outperforms CLIP, ALIGN, and Florence despite their larger image-text pretraining sets.
- Zero-shot Video-Language Tasks: mPLUG is evaluated on video-text retrieval, video captioning, and video question answering using uniformly sampled video frames.
- Zero-shot Video-Language Tasks: Zero-shot mPLUG outperforms models trained with substantially more data, including Florence and BLIP, and some supervised video models.The video-text retrieval comparison includes models using no video pretraining or supervision and no temporal information for the supervised baselines cited.
6 Conclusion
mPLUG is an effective and efficient vision-language pretraining framework for understanding and generation that uses asymmetric cross-modal skip-connections to address information asymmetry and computational inefficiency.
- mPLUG combines an asymmetric vision-language architecture with cross-modal skip-connections for cross-modal understanding and generation.
- The framework is pretrained on large-scale image-text pairs and achieves state-of-the-art performance across a wide range of vision-language tasks.
- mPLUG demonstrates strong zero-shot transferability when directly applied to multiple video-language tasks.
7 More Experiments Details
mPLUG is evaluated across six downstream vision-language tasks using standardized fine-tuning practices and task-specific data, resolutions, and training procedures. The experiments cover VQA, image captioning, image-text retrieval, visual grounding, NLVR2, and SNLI-VE, with pretraining statistics reported separately.
- Evaluation Tasks: Six downstream vision-language tasks are evaluated: VQA, image captioning, image-text retrieval, visual grounding, NLVR2, and SNLI-VE.The evaluation uses official or widely used dataset splits where specified.
- General Settings: Downstream fine-tuning generally uses RandAugment, AdamW with 0.05 weight decay, cosine learning-rate scheduling, and 336 × 336 images.VQA instead uses 504 × 504 images.
- General Settings: VQA and image captioning receive additional continued pretraining on 4M image-text pairs, reported to bring about 0.2+ accuracy improvement.This additional stage is applied before task-specific evaluation.
- Task Procedures: Image captioning uses COCO Karpathy training data, evaluates on COCO Karpathy test and No-Caps validation data, and applies cross-entropy followed by CIDEr optimization.The two-stage fine-tuning lasts five epochs per stage, with learning rates of 1e-5 and 8e-7, respectively.
- Task Procedures: Image-text retrieval uses Karpathy splits for COCO and Flickr30K, while visual grounding evaluates RefCOCO, RefCOCO+, and RefCOCOg.Visual grounding first trains on a mixed dataset, then fine-tunes separately on each dataset.
- Data and Configuration: The pretraining corpus contains 14M image-text examples, with its dataset statistics summarized in Table 12.Fine-tuning hyperparameters for downstream tasks are summarized in Table 11.