Align before Fuse: Vision and Language Representation Learning with Momentum Distillation

1 Introduction

• Existing VLP methods such as LXMERT, UNITER [2], and OSCAR fuse detector-based image regions with words, but face representation misalignment, costly box annotations and high-resolution inference, and noisy web supervision.The cited web datasets can cause objectives such as MLM to overfit noisy text and harm generalization.Source: §1 · p. 1
• ALBEF uses detector-free image and text encoders before cross-modal fusion; its ITC loss aligns representations, strengthens unimodal semantics, and enables informative hard-negative mining.The shared low-dimensional image-text space helps the image-text matching objective select more informative samples.Source: §1 · pp. 1–2
• Momentum distillation uses moving-average model parameters to generate pseudo-targets as additional supervision, helping learning from noisy web annotations and improving pre-training and clean-annotation downstream tasks.It avoids penalizing outputs that are reasonable but differ from web annotations.Source: §1 · p. 2
• The mutual-information account treats ITC and MLM as maximizing a lower bound across partial-information views, while momentum distillation generates views from semantically similar samples, promoting invariance to meaning-preserving transformations.Source: §1 · p. 2
• ALBEF improves retrieval, VQA, visual reasoning, entailment, and weakly supervised grounding; it exceeds larger-data CLIP [6] and ALIGN [7] on retrieval and improves VQA/NLVR2 by 2.37%/3.84% over VILLA, with faster inference.Grad-CAM analysis indicates implicit object, attribute, and relationship grounding.Source: §1 · p. 2

2.1 Vision-Language Representation Learning

• Multimodal transformer encoders support complex reasoning but typically require high-resolution images and pre-trained object detectors [13] [14] [2] [16] [17] [18].A detector-free method [21] improves inference speed but yields lower performance.Source: §2.1 · p. 2
• CLIP [6] and ALIGN [7] use contrastive pre-training on massive noisy web data and excel at retrieval, but lack complex image-text interaction modeling for other V+L tasks [21].These methods learn separate unimodal image and text encoders.Source: §2.1 · p. 2
• ALBEF unifies unimodal and multimodal representations, achieving strong performance on retrieval and reasoning without object detectors [2] [17].Object detectors are a major computation bottleneck for many existing methods.Source: §2.1 · p. 2

2.2 Knowledge Distillation

• Momentum distillation uses a temporal ensemble of the student as its teacher, forming online self-distillation and a generic learning algorithm for improving performance across V+L tasks.Unlike prior work, the paper theoretically and experimentally shows its generic benefits; related ideas have appeared in semi-supervised learning, label-noise learning, and contrastive learning [37].Source: §2.2 · p. 2

3.1 Model Architecture

• ALBEF uses separate image and text encoders, then fuses their features through cross-attention in a multimodal encoder.The image encoder is a 12-layer ViT-B/16; the text and multimodal encoders each use six transformer layers initialized from BERTbase.Source: §3.1 · p. 3, Figure 1 · p. 3

3.2 Pre-training Objectives

• Pre-training combines image-text contrastive learning on unimodal encoders with masked language modeling and image-text matching on the multimodal encoder.MLM masks 15% of input tokens; ITM uses the multimodal [CLS] representation and samples in-batch hard negatives according to contrastive similarity, at zero computational overhead.Source: §3.2 · pp. 3–4, Figure 1 · p. 3

3.3 Momentum Distillation

• Momentum distillation addresses noisy web labels by training the base model to match soft pseudo-targets from exponential-moving-average unimodal and multimodal encoders.The pseudo-targets are used for ITC and MLM; the paper illustrates candidates that can capture relevant image concepts beyond the ground-truth text, and sets α = 0.4 for pre-training and downstream tasks.Source: §3.3 · pp. 4–5

3.4 Pre-training Datasets

• Pre-training uses Conceptual Captions, SBU Captions, COCO, and Visual Genome, totaling 4.0M unique images and 5.1M image-text pairs.Adding the noisier Conceptual 12M dataset raises the total image count to 14.1M.Source: §3.4 · p. 5

3.5 Implementation Details

• The model is pre-trained for 30 epochs at batch size 512 on eight NVIDIA A100 GPUs, using 256 × 256 crops and a momentum parameter of 0.995.Fine-tuning uses 384 × 384 resolution; the contrastive queue contains 65,536 entries, and the distillation weight ramps from 0 to 0.4 during the first epoch.Source: §3.5 · p. 5

4 A Mutual Information Maximization Perspective

• ALBEF seeks view-invariant representations by maximizing mutual information between semantic variations of an image-text pair, using InfoNCE to maximize a lower bound.The views are variations that capture the pair’s semantic meaning; InfoNCE uses positive and negative samples.Source: §4 · p. 5
• ITC treats the image and text modalities as two views and maximizes their mutual information for positive pairs through a symmetric InfoNCE objective.This trains the unimodal encoders to align the two modalities.Source: §4 · p. 5
• MLM maximizes mutual information between a masked word and its context, treating the word and the image plus masked text as the two views [48].The multimodal encoder’s output for the masked context is used to represent that context.Source: §4 · p. 6
• Momentum distillation generates alternative views from the proposal distribution, maximizing mutual information for semantically similar texts, images, and masked-word alternatives.These diverse views augment the original views and encourage the base model to capture view-invariant semantic information.Source: §4 · p. 6

Image-Text Retrieval

• For image-text retrieval, ALBEF jointly optimizes ITC and ITM, then ranks top-k candidates by ITM after scoring all pairs with ITC, enabling faster inference than scoring every pair with ITM [2].ITC labels account for multiple positive texts per image; zero-shot Flickr30K retrieval uses a model fine-tuned on COCO.Source: §5 · p. 6

Visual Entailment

• For visual entailment, ALBEF predicts entailment, neutral, or contradiction using an MLP over the multimodal encoder’s [CLS] representation, treating the task as three-way classification.SNLI-VE test results warrant caution because the test data has been reported as noisy [50].Source: §5 · p. 6

Visual Question Answering

• For VQA, ALBEF generates answers with a six-layer autoregressive transformer decoder that cross-attends to image-question embeddings.The decoder is initialized from multimodal-encoder weights, trained with conditional language modeling, and restricted to 3,192 candidate answers at inference.Source: §5 · pp. 6–7

Natural Language for Visual Reasoning (NLVR2)

• For NLVR2, ALBEF duplicates each multimodal-encoder layer into two blocks to process the image pair and adds an MLP classifier over the [CLS] representation.The blocks share pretrained initialization and cross-attention key/value projections; an additional one-epoch text-assignment pretraining task uses 4M images.Source: §5 · p. 7

Visual Grounding

• For visual grounding, ALBEF fine-tunes on image-text supervision without bounding-box annotations and uses Grad-CAM heatmaps to rank detected proposals.Experiments use RefCOCO+ [56].Source: §5 · p. 7

6.1 Evaluation on the Proposed Methods

• Adding ITC improves performance across tasks; hard-negative mining improves ITM, while momentum distillation improves ITC, MLM, and all downstream tasks.The last-row variant shows that ALBEF can use more noisy web data to improve pre-training performance.Source: §6.1 · pp. 7–8

6.2 Evaluation on Image-Text Retrieval

• ALBEF achieves state-of-the-art fine-tuned and zero-shot image-text retrieval, outperforming CLIP [6] and ALIGN [7] despite their much larger training datasets.The paper reports considerable improvement as ALBEF's training images increase from 4M to 14M and hypothesizes further growth with larger-scale web image-text pairs.Source: Table 2 · p. 8, Table 3 · p. 8, §6.2 · p. 8

6.3 Evaluation on VQA, NLVR, and VE

• 2.37%, 3.84%, and 1.88% absolute improvements over VILLA on VQA test-std, NLVR2 test-P, and SNLI-VE test, respectively; ALBEF is also >10 times faster on NLVR2.With 14M pre-training images, ALBEF substantially outperforms existing methods, including methods using object tags or adversarial data augmentation; it is detector-free and uses lower-resolution images.Source: Table 4 · p. 8, §6.3 · p. 8

6.4 Weakly-supervised Visual Grounding

• On RefCOCO+, ALBEF substantially outperforms methods, whose text embeddings are weaker; Grad-CAM visualizations show grounding of objects, attributes, and relationships.ALBEFitc uses gradients from image-text similarity sitc on the last-layer image self-attention maps, while ALBEFitm uses image-text matching sitm on third-layer multimodal cross-attention maps. VQA Grad-CAM maps are described as highly correlated with where humans look.Source: §6.4 · pp. 8–9, Table 5 · p. 9

6.5 Ablation Study

• In the retrieval ablation, sitm ranking is generally insensitive to k when sitc filters top-k candidates; the final column evaluates hard-negative mining.The ablation reports average recall on the test set.Source: §6.5 · p. 9, Table 6 · p. 9

• ALBEF aligns unimodal image and text representations before multimodal fusion, and the paper verifies the effectiveness of image-text contrastive learning and momentum distillation theoretically and experimentally.Source: §7 · pp. 9–10
• ALBEF offers better performance and faster inference than existing methods on multiple downstream vision-language tasks.Source: §7 · p. 10
• Further data and model analysis is needed before deployment because web data may include private information, unsuitable images, or harmful texts, and accuracy-only optimization may have unwanted social implications.Source: §7 · p. 10

• All downstream tasks use 384 × 384 images and the pre-training RandAugment, AdamW, cosine learning-rate decay, weight decay, and distillation weight; inference resizes images without cropping.Source: A Downstream Task Details · p. 13

Image-Text Retrieval

• Image-text retrieval uses the Karpathy split [59] of COCO and Flickr30K, with 10 epochs of fine-tuning.COCO has 113/5k/5k train/validation/test examples and Flickr30K has 29k/1k/1k images; the batch size is 256 and the initial learning rate is 1e−5.Source: A Downstream Task Details · p. 13

Visual Entailment

• Visual entailment is evaluated on SNLI-VE [51], constructed from SNLI and Flickr30K, using the original dataset split.The split has 29.8k training images and 1k each for evaluation and test; fine-tuning runs for 5 epochs with batch size 256 and initial learning rate 2e−5.Source: A Downstream Task Details · p. 13

VQA

• For VQA2.0, training uses the train and validation sets plus Visual Genome question-answer pairs, with answer losses weighted by each answer’s occurrence percentage.The model is evaluated on test-dev and test-std and fine-tuned for 8 epochs.Source: A Downstream Task Details · p. 13

NLVR2

• NLVR2 fine-tuning follows the original train/validation/test split, with an additional text-assignment pre-training step to prepare the model to reason over two images.Text-assignment pre-training uses 256 × 256 images from the 4M dataset for 1 epoch; fine-tuning runs for 10 epochs.Source: A Downstream Task Details · p. 13

Visual Grounding

• On RefCOCO+ [56], the model was not allowed to see validation/test images but had encountered them during pre-training; authors hypothesize little effect because these images are a very small portion of the 14M pre-training images.Source: A Downstream Task Details · p. 13

B Additional Per-word Visualizations

• Figure 9 visualizes per-word Grad-CAM to demonstrate grounding of objects, actions, attributes, and relationships.Source: B Additional Per-word Visualizations · p. 14
• Figure 10 compares human attention with ALBEF’s Grad-CAM for VQA.Source: Figure 10 · p. 15

C Comparison with Human Attention

• The rank correlation between ALBEF VQA Grad-CAM and human attention maps for ground-truth answers was 0.205 across 1,374 validation question-image pairs.The result suggests ALBEF attends to appropriate regions despite not being trained on grounded image-text pairs.Source: C Comparison with Human Attention · p. 15
• Human attention maps were collected by asking Amazon Mechanical Turk participants to deblur image regions until they could confidently answer VQA questions.The collection used questions and blurred versions of images from a subset of the VQA dataset.Source: C Comparison with Human Attention · p. 15
• The comparison used Grad-CAM from ALBEF’s third multimodal cross-attention layer and the rank-correlation protocol from.Grad-CAM visualizations were computed for 1,374 validation question-image pairs.Source: C Comparison with Human Attention · p. 15
• Qualitative examples compare human attention maps with ALBEF VQA Grad-CAM visualizations in Figure 10.The visual comparison is described for ground-truth answers.Source: C Comparison with Human Attention · p. 15

D Additional Examples of Pseudo-targets

• Figure 11 presents examples of the five most similar texts selected by the momentum model for ITC.The figure illustrates pseudo-target examples for image-text contrastive learning.Source: Figure 11 · p. 15
• Figure 12 presents examples of the five top words generated by the momentum model for MLM.The figure illustrates pseudo-target examples for masked language modeling.Source: Figure 12 · p. 16

E Pre-training Dataset Details

• Table 8 reports image and text statistics for the pre-training datasets.The supplied passages identify the table’s scope but do not provide its statistics.Source: E Pre-training Dataset Details · p. 16, Table 8 · p. 16