Source-linked AI summary
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, Steven Hoi
TL;DR
Existing vision-language pretraining struggles with misaligned image and text representations, costly detector-based features, and noisy web supervision. ALBEF aligns modalities before fusion and adds momentum distillation, achieving strong performance across downstream tasks, including 2.37% and 3.84% absolute gains on VQA and NLVR2 versus VILLA.
Problem
Existing VLP methods face misaligned modalities, detector and high-resolution image costs, and noisy web labels that can penalize valid predictions.
Method
ALBEF aligns independently encoded image and text representations with contrastive learning before multimodal fusion, while momentum distillation supplies pseudo-targets from a moving-average model.
Results
2.37% and 3.84% absolute improvements are reported on VQA and NLVR2 versus VILLA, with faster inference and strong performance across multiple downstream tasks.
Takeaways & Limitations
ALBEF provides a detector-free framework that combines aligned unimodal and multimodal representations for retrieval and reasoning tasks.
Takeaways & Limitations
The authors state that further data and model analysis is needed before deployment because web data may contain private information, unsuitable images, or harmful texts.
Abstract
from arXiv · showhide
Large-scale vision and language representation learning has shown promising improvements on various vision-language tasks. Most existing methods employ a transformer-based multimodal encoder to jointly model visual tokens (region-based image features) and word tokens. Because the visual tokens and word tokens are unaligned, it is challenging for the multimodal encoder to learn image-text interactions. In this paper, we introduce a contrastive loss to ALign the image and text representations BEfore Fusing (ALBEF) them through cross-modal attention, which enables more grounded vision and language representation learning. Unlike most existing methods, our method does not require bounding box annotations nor high-resolution images. In order to improve learning from noisy web data, we propose momentum distillation, a self-training method which learns from pseudo-targets produced by a momentum model. We provide a theoretical analysis of ALBEF from a mutual information maximization perspective, showing that different training tasks can be interpreted as different ways to generate views for an image-text pair. ALBEF achieves state-of-the-art performance on multiple downstream vision-language tasks. On image-text retrieval, ALBEF outperforms methods that are pre-trained on orders of magnitude larger datasets. On VQA and NLVR$^2$, ALBEF achieves absolute improvements of 2.37% and 3.84% compared to the state-of-the-art, while enjoying faster inference speed. Code and pre-trained models are available at https://github.com/salesforce/ALBEF/.
1 Introduction
ALBEF addresses misaligned image-text representations, detector and resolution costs, and noisy web supervision by aligning modalities before fusion and using momentum distillation. It achieves strong results across downstream vision-language tasks, including retrieval, VQA, and NLVR2.
- Existing VLP methods fuse detector-based image features with word tokens, but their representations occupy separate spaces, hindering image-text interaction learning.These methods commonly train multimodal encoders with masked language modeling and image-text matching.
- ALBEF independently encodes images and text, applies image-text contrastive learning before cross-modal fusion, and uses a detector-free image encoder.The contrastive objective aligns modalities and supports a shared low-dimensional embedding space for harder negative mining.
- Momentum distillation uses a moving-average model to generate pseudo-targets, reducing penalties for reasonable outputs that differ from noisy web annotations.The method improves pre-training and downstream tasks with clean annotations.
- ALBEF interprets ITC and MLM as mutual-information lower-bound objectives over different views of image-text pairs, with momentum distillation generating semantically similar views.The resulting representations are described as invariant to semantic-preserving transformations.
- 2.37% and 3.84% absolute improvements are reported on VQA and NLVR2 versus VILLA, alongside faster inference speed and strong retrieval performance.On image-text retrieval, ALBEF outperforms methods pretrained on orders of magnitude larger datasets.
2 Related Work
ALBEF combines strong unimodal and multimodal representation learning, targeting both retrieval and reasoning while avoiding object detectors. Its momentum distillation is framed as online self-distillation and as a broadly applicable learning algorithm for vision-language tasks.
- Multimodal representation learning: Transformer-based multimodal methods model image-text interactions effectively but commonly require high-resolution images and pretrained object detectors.
- ALBEF: ALBEF unifies unimodal and multimodal representation learning, achieving strong performance on both retrieval and reasoning tasks without object detectors.This removes a major computational bottleneck in many existing methods.
- Knowledge distillation: Momentum distillation is interpreted as online self-distillation, using a temporal ensemble of the student model as its teacher.The paper distinguishes this approach from methods that distill from a fixed pretrained teacher and reports benefits across many vision-language tasks.
3 ALBEF Pre-training
ALBEF pre-trains separate image and text encoders before multimodal fusion, combining contrastive, matching, and masked-language objectives with momentum-generated pseudo-targets to handle noisy web supervision.
- Model architecture: ALBEF uses image, text, and multimodal encoders, with cross-modal attention fusing image features and text features.The image and text encoders independently produce representations before the multimodal encoder models their interactions.
- Pre-training objectives: ALBEF pre-trains with ITC on unimodal encoders and MLM and ITM on the multimodal encoder.The full objective is L = L_itc + L_mlm + L_itm, with ITM enhanced by online contrastive hard-negative mining.
- Image-text contrastive learning: Image-text contrastive learning aligns unimodal representations before fusion, learns a shared 256-d space, and supports contrastive hard-negative mining for image-text matching.The similarity function compares normalized projected [CLS] embeddings, while in-batch negatives are sampled according to contrastive similarity.
- Pre-training objectives: MLM masks 15% of input tokens and predicts them using both the image and contextual text, while ITM classifies image-text pairs as matched or unmatched.ITM uses the multimodal [CLS] representation and a two-class softmax prediction.
- Momentum distillation: Momentum distillation uses a continuously evolving moving-average teacher to generate pseudo-targets that supervise the base model alongside standard losses.The method addresses noisy web pairs whose one-hot labels can penalize alternative correct matches or words, and applies a weighted KL-divergence term with α = 0.4.
4 A Mutual Information Maximization Perspective
ALBEF interprets its training objectives as maximizing mutual information between different views of an image-text pair. ITC and MLM use partial views, while momentum distillation generates semantically similar alternative views.
- ITC, MLM, and momentum distillation are framed as different ways to generate views of an image-text pair for mutual-information maximization.The goal is to learn representations invariant to changes in view.
- ITC treats the image and text modalities as two views and maximizes their mutual information for positive pairs through a symmetric InfoNCE objective.The loss trains unimodal encoders to associate matching image-text representations.
- MLM generates views by pairing a randomly selected word token with the image and text context in which that word is masked.This corresponds to maximizing mutual information between the masked token and its masked context.
- Momentum distillation generates alternative views from the proposal distribution, including semantically similar texts or images and alternative masked-word targets.These views act as data augmentation beyond the original image-text pairs.
5 Downstream V+L Tasks
The paper adapts ALBEF to retrieval, visual entailment, VQA, NLVR2, and weakly supervised visual grounding with task-specific fine-tuning strategies. VQA uses answer generation, while NLVR2 modifies the multimodal encoder for two-image reasoning.
- Downstream Tasks: ALBEF is adapted to five downstream vision-language tasks using task-specific fine-tuning strategies.The tasks include image-text retrieval, visual entailment, VQA, NLVR2, and weakly supervised visual grounding.
- Image-Text Retrieval: Image-text retrieval evaluates image-to-text and text-to-image retrieval on Flickr30K and COCO while jointly optimizing ITC and ITM losses.ITC provides an image-text scoring function based on unimodal representation similarity.
- Visual Entailment: Visual entailment predicts whether an image-text relationship is entailment, neutral, or contradiction using the multimodal encoder’s [CLS] representation.The task is formulated as three-way classification.
- Visual Question Answering: VQA formulates answer prediction as generation with a six-layer autoregressive transformer decoder receiving multimodal embeddings through cross-attention.The decoder is initialized from the multimodal encoder and trained with conditional language modeling.
- Caveat: SNLI-VE results should be interpreted cautiously because the test data has been reported to be noisy.
- Natural Language for Visual Reasoning: NLVR2 extends the multimodal encoder with replicated transformer blocks so it can reason over a pair of images.An additional text-assignment pre-training task assigns text to the first image, second image, or neither.
6 Experiments
ALBEF's ablations and benchmark evaluations show that ITC, hard-negative mining, and momentum distillation improve performance across vision-language tasks. The detector-free model achieves state-of-the-art retrieval, reasoning, and weakly supervised grounding results, with faster inference.
- Evaluation of the Proposed Methods: Adding ITC to MLM+ITM substantially improves the pre-trained model across all evaluated tasks.
- Evaluation of the Proposed Methods: Momentum distillation improves ITC, MLM, and all downstream tasks while enabling effective use of noisier web data.
- Image-Text Retrieval: ALBEF achieves state-of-the-art fine-tuned and zero-shot image-text retrieval, outperforming CLIP and ALIGN despite their much larger pre-training datasets.
- VQA, NLVR, and VE: 2.37% on VQA test-std, 3.84% on NLVR2 test-P, and 1.88% on SNLI-VE test are ALBEF's absolute improvements over VILLA.
- Weakly-supervised Visual Grounding: ALBEF substantially outperforms existing methods on weakly supervised RefCOCO+ visual grounding without bounding-box annotations.
- Weakly-supervised Visual Grounding: Grad-CAM visualizations indicate grounding of objects, attributes, and relationships, while NLVR2 ranking remains generally insensitive to the top-k candidate count.
7 Conclusion and Social Impacts
The paper concludes that ALBEF combines strong performance with faster inference across multiple vision-language tasks. It also cautions that further analysis is needed before deployment because web data and accuracy-only optimization may create social risks.
- Conclusion: ALBEF combines image-text contrastive learning and momentum distillation, achieving better performance and faster inference across multiple downstream tasks.
- Social Impacts: Deployment requires additional data and model analysis because web data may contain private information, unsuitable images, or harmful texts.
- Social Impacts: Optimizing accuracy alone may have unwanted social implications, limiting the paper's deployment conclusions.
A Downstream Task Details
Downstream evaluations use standardized fine-tuning settings across tasks, with task-specific datasets and procedures. Visual grounding additionally uses Grad-CAM-based patch importance from image or multimodal attention.
- Fine-tuning setup: All downstream tasks use 384 × 384 input images, resizing without cropping during inference.The implementation reuses RandAugment, AdamW, cosine decay, weight decay, and distillation weight from pre-training.
- Image-Text Retrieval: Image-text retrieval is evaluated on COCO and Flickr30K using the Karpathy split and 10-epoch fine-tuning.COCO and Flickr30K use batch size 256 and an initial learning rate of 1e−5.
- Visual Entailment: Visual entailment is evaluated on SNLI-VE with the original split and 5-epoch fine-tuning.The task uses batch size 256 and an initial learning rate of 2e−5.
- NLVR2: NLVR2 uses an additional text-assignment pre-training stage because each example contains two input images.Text-assignment pre-training runs for 1 epoch on the 4M dataset with 256 × 256 images.
- Visual Grounding: Visual grounding on RefCOCO+ computes patch importance with Grad-CAM from either visual self-attention or multimodal cross-attention.For ITM, Grad-CAM uses the third multimodal layer and averages across heads and input text tokens.
B Additional Per-word Visualizations
ALBEF’s per-word Grad-CAM visualizations are used to examine fine-grained visual grounding. The accompanying figure presents Grad-CAM visualizations on cross-attention maps for individual words.
- Additional Per-word Visualizations: Per-word Grad-CAM visualizations demonstrate grounding of objects, actions, attributes, and relationships.The visualizations are presented as additional evidence of the model’s visual-grounding ability.
- Additional Per-word Visualizations: Figure 10 compares human attention with ALBEF’s Grad-CAM for VQA.
C Comparison with Human Attention
The paper compares ALBEF’s VQA Grad-CAM visualizations with human attention maps and illustrates pseudo-targets generated by momentum models for contrastive and masked-language objectives.
- Comparison with Human Attention: Human attention maps are compared with ALBEF VQA Grad-CAM on 1,374 validation question-image pairs using rank correlation.The Grad-CAM is computed at the third multimodal cross-attention layer.
- Momentum-model examples: Pseudo-target examples include semantically related descriptions for demonstrations, rock-star performances, and snowy houses with fences.
- Momentum-model examples: The examples show top-5 similar texts selected by the momentum model for image-text contrastive learning.
- Momentum-model examples: The examples show top-5 words generated by the momentum model for masked language modeling.
E Pre-training Dataset Details
The paper reports statistics for the image and text components of its pre-training datasets in Table 8.
- Pre-training Dataset Details: Table 8 provides statistics for the image data used in pre-training.
- Pre-training Dataset Details: Table 8 provides statistics for the text data used in pre-training.
- Pre-training Dataset Details: The pre-training dataset statistics are organized together in Table 8.