VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts

Motivation

• Dual encoders efficiently support retrieval but provide shallow image-text interaction, limiting complex vision-language classification tasks.Fusion encoders model deeper cross-modal interactions for classification.Source: §1 · p. 1

Unified model

• VLMO uses a shared MOME Transformer as either a dual encoder for retrieval or a fusion encoder for classification.The architecture switches between separate and joint image-text encoding.Source: §1 · pp. 1–2

Architecture

• MOME replaces the standard feed-forward network with modality-specific experts and retains shared self-attention across modalities.Its experts separately encode vision, language, and vision-language inputs.Source: §1 · p. 2

Pre-training

• Stagewise pre-training first uses image-only masked image modeling, then text-only masked language modeling, before vision-language pre-training.This strategy leverages large-scale unimodal corpora in addition to image-text pairs.Source: §1 · p. 2

Results

• VLMO achieves state-of-the-art results on vision-language retrieval and classification tasks, including VQA and NLVR2.As a dual encoder, it also reports faster retrieval inference than fusion-encoder-based models.Source: §1 · p. 2

Background

• Transformer pre-training has advanced natural language processing, computer vision, and vision-language tasks.The related work situates vision-language pre-training within broader Transformer-based progress.Source: §2 · p. 2

Dual encoders

• Dual encoders separately encode images and text and are effective for retrieval, but their simple interaction is insufficient for some complex tasks.They commonly use cosine similarity or a linear projection to model image-text interaction.Source: §2 · p. 2

Fusion encoders

• Fusion encoders concatenate image patch and word embeddings before Transformer processing to model contextualized cross-modal representations.This approach jointly processes image and text rather than relying only on separate feature vectors.Source: §2 · p. 3

VLMO

• VLMO's shared MOME Transformer enables separate encoding for retrieval and joint encoding for deeper interaction in classification.The paper reports competitive performance with faster inference for both retrieval and classification.Source: §2 · p. 3

Unified architecture and pre-training

• VLMO obtains image-only, text-only, and image-text pair representations with a shared MOME Transformer for unified pre-training.The shared model is optimized with contrastive learning, image-text matching, and masked language modeling across the corresponding representations.Source: §3 · p. 3, §3.3 · p. 5

Stagewise pre-training

• Stagewise pre-training first trains vision components on image-only data, freezes them while training the language expert on text-only data, then trains the whole model on vision-language data.The vision stage uses masked image modeling, and the language stage uses masked language modeling.Source: Figure 2 · p. 4, §3.4 · p. 6

Input representations

• Image representations split an image into N = HW/P^2 patches, linearly project them, prepend [I_CLS], and add position and type embeddings.The image input is formed from patch embeddings and a learnable special token.Source: §3.1 · p. 4
• Text representations tokenize subwords with WordPiece, add [T_CLS] and [T_SEP], and sum word, position, and type embeddings.The resulting sequence represents the tokenized text input for the Transformer.Source: §3.1 · p. 4

Mixture-of-Modality-Experts Transformer

• MoME replaces the standard feed-forward network with modality experts while retaining self-attention shared across modalities.The model uses vision, language, and vision-language experts; mixed image-text inputs use modality-specific experts at lower layers and the vision-language expert at top layers.Source: §3.2 · pp. 4–5

Task adaptation

• VLMO uses a dual encoder with contrastive optimization for retrieval and a fusion encoder with a [T_CLS]-based classifier for vision-language classification.Separate encoding enables faster retrieval inference, while fusion encoding models image-text interaction for tasks such as VQA and visual reasoning.Source: Figure 3 · p. 5, §3.5 · p. 6

Experimental Setup

• The experiments evaluate VLMO on vision-language classification and retrieval, using large-scale image-text pre-training data and downstream fine-tuning.The setup uses four captioning datasets with about 4M images and 10M image-text pairs, alongside a one-billion-pair scaling experiment.Source: §4 · p. 6, §4.1 · p. 6, §4.2 · p. 7

Vision-Language Classification

• VLMO is fine-tuned as a fusion encoder for VQA and NLVR2 classification tasks.VQA predicts answers from image-question pairs, while NLVR2 predicts whether a description is true for a pair of images.Source: §4.3 · p. 7
• VLMO achieves state-of-the-art vision-language classification performance, with the large model surpassing larger SimVLM-Huge and Florence-Huge models.The model uses linear image projection, providing a significant speedup over approaches based on detector-extracted image region features.Source: Table 1 · p. 7, §4.3 · p. 8

Vision-Language Retrieval

• VLMO achieves competitive retrieval performance while offering much faster inference than fusion-encoder-based models.Fusion encoders require jointly encoding all image-text pairs, whereas VLMO separately encodes images and text for similarity computation.Source: §4.4 · p. 8, Table 2 · p. 8

Vision Tasks

• As an image-only encoder, VLMO achieves competitive ImageNet classification and ADE20K semantic segmentation results, slightly exceeding BEIT initialization.These evaluations use 224×224 images for ImageNet and 512×512 images for ADE20K.Source: §4.5 · p. 8, Table 3 · p. 9

Ablation Studies

• Ablations show benefits from stagewise pre-training, MoME Transformer components, unified pre-training tasks, and global hard-negative mining.Unified training outperforms image-text contrastive learning alone, image-text matching improves results, masked language modeling contributes positively, and global hard-negative mining brings significant improvements.Source: §4.6 · p. 9

• VLMO jointly learns dual-encoder and fusion-encoder capabilities with a shared MOME Transformer backbone.It supports efficient retrieval through separate encoding and classification through cross-modal interaction.Source: §5 · p. 10
• MOME uses modality experts and shared self-attention to encode modality-specific information and align different modalities.Source: §5 · p. 10
• Stagewise pre-training leverages large-scale image-only and text-only corpora in addition to image-text pairs.Source: §5 · p. 10
• VLMO outperforms previous state-of-the-art models on various vision-language classification and retrieval benchmarks.Source: §5 · p. 10
• Future work includes scaling model size, supporting vision-language generation, studying cross-modal benefits, and integrating speech, video, or structured knowledge.Source: §5 · p. 10

• Shared self-attention achieves better performance than separate self-attention in the MOME ablation.Separate attention uses different parameters for image patches and text tokens in the first L−F layers.Source: A Ablation Study of Shared… · p. 16, Table 7 · p. 16
• Shared self-attention helps VLMO learn modality alignment and fuse images with text in lower layers for classification.Source: A Ablation Study of Shared… · p. 16

• English Wikipedia and BookCorpus provide the text-only pre-training data.Source: B Hyperparameters for Text-Only… · p. 16
• AdamW uses β1 = 0.9, β2 = 0.98, with maximum sequence length 196, batch size 1024, and weight decay 0.01.Source: B Hyperparameters for Text-Only… · p. 16
• The peak learning rate is 2e-4 for text-only pre-training.Source: B Hyperparameters for Text-Only… · p. 16
• The base-size model trains for 500k steps, while the large-size model trains for 200k steps.Source: B Hyperparameters for Text-Only… · p. 16

Visual Question Answering (VQA)

• VQA fine-tuning runs for 10 epochs with batch size 128 and image resolution 480 × 480.The peak learning rate is 3e-5 for base-size and 1.5e-5 for large-size models; VLMO-Large++ uses 768 × 768 resolution.Source: C Hyperparameters for Vision-Language… · p. 16

Natural Language for Visual Reasoning (NLVR2)

• NLVR2 fine-tuning runs for 10 epochs with batch size 128 and image resolution 384 × 384.Peak learning rates are 5e-5 for base-size and 3e-5 for large-size models.Source: C Hyperparameters for Vision-Language… · p. 16
• NLVR2 ablations use 224 × 224 resolution and select learning rates from {5e-5, 1e-4}.The ablation results are averaged over 3 runs.Source: C Hyperparameters for Vision-Language… · p. 16

• COCO models use 20 epochs for base size and 10 for large size, with batch size 2048 and resolution 384 × 384.Peak learning rates are 2e-5 for base size and 1e-5 for large size.Source: D Hyperparameters for Vision-Language… · p. 16
• Flickr30K models use 40 epochs, batch size 2048, peak learning rate 1e-5, and resolution 384 × 384.The fine-tuned COCO model initializes Flickr30K training.Source: D Hyperparameters for Vision-Language… · p. 16
• Flickr30K ablations use 10 epochs, batch size 1024, peak learning rate 5e-5, and resolution 224 × 224.Source: D Hyperparameters for Vision-Language… · p. 16