Source-linked AI summary
VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts
Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Furu Wei
TL;DR
Vision-language pre-training must support both efficient retrieval and deep cross-modal classification, but prior architectures typically specialize in one mode. VLMO introduces a shared MOME Transformer and stagewise pre-training over multimodal and unimodal data, achieving state-of-the-art results across retrieval and classification tasks.
Problem
Prior dual encoders support efficient retrieval but have shallow interaction, while fusion encoders handle deeper interaction without providing the same unified architecture.
Method
VLMO uses modality-specific experts with shared self-attention and stagewise pre-training on image-only, text-only, and image-text data.
Results
VLMO achieves state-of-the-art results across vision-language retrieval and classification, including VQA and NLVR2.
Takeaways & Limitations
The shared MOME backbone supports dual-encoder retrieval and fusion-encoder classification within one pretrained vision-language model.
Takeaways & Limitations
The paper identifies extending VLMO to additional modalities such as speech, video, and structured knowledge as future work.
Abstract
from arXiv · showhide
We present a unified Vision-Language pretrained Model (VLMo) that jointly learns a dual encoder and a fusion encoder with a modular Transformer network. Specifically, we introduce Mixture-of-Modality-Experts (MoME) Transformer, where each block contains a pool of modality-specific experts and a shared self-attention layer. Because of the modeling flexibility of MoME, pretrained VLMo can be fine-tuned as a fusion encoder for vision-language classification tasks, or used as a dual encoder for efficient image-text retrieval. Moreover, we propose a stagewise pre-training strategy, which effectively leverages large-scale image-only and text-only data besides image-text pairs. Experimental results show that VLMo achieves state-of-the-art results on various vision-language tasks, including VQA, NLVR2 and image-text retrieval. The code and pretrained models are available at https://aka.ms/vlmo.
1 Introduction
VLMO unifies dual-encoder retrieval and fusion-encoder classification within a shared Mixture-of-Modality-Experts Transformer. Stagewise pre-training further uses image-only and text-only data, and experiments report state-of-the-art vision-language results.
- Motivation: Dual encoders efficiently support retrieval but provide shallow image-text interaction, limiting complex vision-language classification tasks.Fusion encoders model deeper cross-modal interactions for classification.
- Unified model: VLMO uses a shared MOME Transformer as either a dual encoder for retrieval or a fusion encoder for classification.The architecture switches between separate and joint image-text encoding.
- Architecture: MOME replaces the standard feed-forward network with modality-specific experts and retains shared self-attention across modalities.Its experts separately encode vision, language, and vision-language inputs.
- Pre-training: Stagewise pre-training first uses image-only masked image modeling, then text-only masked language modeling, before vision-language pre-training.This strategy leverages large-scale unimodal corpora in addition to image-text pairs.
- Results: VLMO achieves state-of-the-art results on vision-language retrieval and classification tasks, including VQA and NLVR2.As a dual encoder, it also reports faster retrieval inference than fusion-encoder-based models.
2 Related Work
Related work centers on separate dual encoders for efficient retrieval and fusion encoders for deeper cross-modal interaction. VLMO addresses this architectural split with shared MOME-based pre-training and supports both encoding modes.
- Background: Transformer pre-training has advanced natural language processing, computer vision, and vision-language tasks.The related work situates vision-language pre-training within broader Transformer-based progress.
- Dual encoders: Dual encoders separately encode images and text and are effective for retrieval, but their simple interaction is insufficient for some complex tasks.They commonly use cosine similarity or a linear projection to model image-text interaction.
- Fusion encoders: Fusion encoders concatenate image patch and word embeddings before Transformer processing to model contextualized cross-modal representations.This approach jointly processes image and text rather than relying only on separate feature vectors.
- VLMO: VLMO's shared MOME Transformer enables separate encoding for retrieval and joint encoding for deeper interaction in classification.The paper reports competitive performance with faster inference for both retrieval and classification.
3 Methods
VLMO uses a unified MoME Transformer and stagewise pre-training to support both separate image-text retrieval and deep image-text classification. Its modality experts, shared attention, and task-specific fine-tuning connect image-only, text-only, and paired representations across the training pipeline.
- Unified architecture and pre-training: VLMO obtains image-only, text-only, and image-text pair representations with a shared MOME Transformer for unified pre-training.The shared model is optimized with contrastive learning, image-text matching, and masked language modeling across the corresponding representations.
- Stagewise pre-training: Stagewise pre-training first trains vision components on image-only data, freezes them while training the language expert on text-only data, then trains the whole model on vision-language data.The vision stage uses masked image modeling, and the language stage uses masked language modeling.
- Input representations: Image representations split an image into N = HW/P^2 patches, linearly project them, prepend [I_CLS], and add position and type embeddings.The image input is formed from patch embeddings and a learnable special token.
- Input representations: Text representations tokenize subwords with WordPiece, add [T_CLS] and [T_SEP], and sum word, position, and type embeddings.The resulting sequence represents the tokenized text input for the Transformer.
- Mixture-of-Modality-Experts Transformer: MoME replaces the standard feed-forward network with modality experts while retaining self-attention shared across modalities.The model uses vision, language, and vision-language experts; mixed image-text inputs use modality-specific experts at lower layers and the vision-language expert at top layers.
- Task adaptation: VLMO uses a dual encoder with contrastive optimization for retrieval and a fusion encoder with a [T_CLS]-based classifier for vision-language classification.Separate encoding enables faster retrieval inference, while fusion encoding models image-text interaction for tasks such as VQA and visual reasoning.
4 Experiments
VLMO is evaluated across vision-language classification, retrieval, image-only vision tasks, and ablations of its data, architecture, and pre-training objectives.
- Experimental Setup: The experiments evaluate VLMO on vision-language classification and retrieval, using large-scale image-text pre-training data and downstream fine-tuning.The setup uses four captioning datasets with about 4M images and 10M image-text pairs, alongside a one-billion-pair scaling experiment.
- Vision-Language Classification: VLMO is fine-tuned as a fusion encoder for VQA and NLVR2 classification tasks.VQA predicts answers from image-question pairs, while NLVR2 predicts whether a description is true for a pair of images.
- Vision-Language Classification: VLMO achieves state-of-the-art vision-language classification performance, with the large model surpassing larger SimVLM-Huge and Florence-Huge models.The model uses linear image projection, providing a significant speedup over approaches based on detector-extracted image region features.
- Vision-Language Retrieval: VLMO achieves competitive retrieval performance while offering much faster inference than fusion-encoder-based models.Fusion encoders require jointly encoding all image-text pairs, whereas VLMO separately encodes images and text for similarity computation.
- Vision Tasks: As an image-only encoder, VLMO achieves competitive ImageNet classification and ADE20K semantic segmentation results, slightly exceeding BEIT initialization.These evaluations use 224×224 images for ImageNet and 512×512 images for ADE20K.
- Ablation Studies: Ablations show benefits from stagewise pre-training, MoME Transformer components, unified pre-training tasks, and global hard-negative mining.Unified training outperforms image-text contrastive learning alone, image-text matching improves results, masked language modeling contributes positively, and global hard-negative mining brings significant improvements.
5 Conclusion
VLMO unifies dual-encoder retrieval and fusion-encoder classification through a shared MOME Transformer, while stagewise pre-training uses image-only and text-only corpora. The authors report improved performance across vision-language benchmarks and identify several future extensions.
- VLMO jointly learns dual-encoder and fusion-encoder capabilities with a shared MOME Transformer backbone.It supports efficient retrieval through separate encoding and classification through cross-modal interaction.
- MOME uses modality experts and shared self-attention to encode modality-specific information and align different modalities.
- Stagewise pre-training leverages large-scale image-only and text-only corpora in addition to image-text pairs.
- VLMO outperforms previous state-of-the-art models on various vision-language classification and retrieval benchmarks.
- Future work includes scaling model size, supporting vision-language generation, studying cross-modal benefits, and integrating speech, video, or structured knowledge.
A Ablation Study of Shared Self-Attention
The ablation study compares shared and separate self-attention in MOME, finding that shared attention improves performance and supports cross-modal alignment and fusion.
- Shared self-attention achieves better performance than separate self-attention in the MOME ablation.Separate attention uses different parameters for image patches and text tokens in the first L−F layers.
- Shared self-attention helps VLMO learn modality alignment and fuse images with text in lower layers for classification.
B Hyperparameters for Text-Only Pre-Training
Text-only pre-training uses English Wikipedia and BookCorpus with AdamW and specified sequence, batch, optimization, and training-step settings.
- English Wikipedia and BookCorpus provide the text-only pre-training data.
- AdamW uses β1 = 0.9, β2 = 0.98, with maximum sequence length 196, batch size 1024, and weight decay 0.01.
- The peak learning rate is 2e-4 for text-only pre-training.
- The base-size model trains for 500k steps, while the large-size model trains for 200k steps.
C Hyperparameters for Vision-Language Classification Fine-Tuning
Vision-language classification fine-tuning uses task-specific training settings for VQA and NLVR2, including different resolutions and learning rates by model size.
- Visual Question Answering (VQA): VQA fine-tuning runs for 10 epochs with batch size 128 and image resolution 480 × 480.The peak learning rate is 3e-5 for base-size and 1.5e-5 for large-size models; VLMO-Large++ uses 768 × 768 resolution.
- Natural Language for Visual Reasoning (NLVR2): NLVR2 fine-tuning runs for 10 epochs with batch size 128 and image resolution 384 × 384.Peak learning rates are 5e-5 for base-size and 3e-5 for large-size models.
- Natural Language for Visual Reasoning (NLVR2): NLVR2 ablations use 224 × 224 resolution and select learning rates from {5e-5, 1e-4}.The ablation results are averaged over 3 runs.
D Hyperparameters for Vision-Language Retrieval Fine-Tuning
Vision-language retrieval fine-tuning uses dataset-specific training schedules and learning rates, with 384 × 384 inputs for COCO and standard Flickr30K results. Flickr30K ablations use shorter training, smaller batches, higher learning rates, and 224 × 224 inputs.
- COCO models use 20 epochs for base size and 10 for large size, with batch size 2048 and resolution 384 × 384.Peak learning rates are 2e-5 for base size and 1e-5 for large size.
- Flickr30K models use 40 epochs, batch size 2048, peak learning rate 1e-5, and resolution 384 × 384.The fine-tuned COCO model initializes Flickr30K training.
- Flickr30K ablations use 10 epochs, batch size 1024, peak learning rate 5e-5, and resolution 224 × 224.