PaperlayerView exact revision on arXiv

Source-linked AI summary

UNITER: UNiversal Image-TExt Representation Learning

Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, Jingjing Liu

arXiv:1909.11740v3cs.CVcs.CLcs.LG

TL;DR

Existing V+L representations are often task-specific, motivating a universal image-text representation for heterogeneous tasks. UNITER introduces new pre-training objectives and achieves state-of-the-art performance across multiple V+L benchmarks, with a noted input-setting boundary for NLVR2.

  • Problem

    Existing multimodal architectures and learned representations are highly task-specific, limiting their generalizability across V+L tasks.

  • Method

    UNITER combines Conditional Masking, Optimal-Transport-based Word-Region Alignment, and masked language and region modeling for universal image-text representation learning.

  • Results

    UNITER achieves new state-of-the-art results across multiple V+L tasks; UNITER-base exceeds prior state of the art by approximately +2.8% on VCR Q→AR, +2.5% on NLVR2, and +7% on SNLI-VE.

  • Takeaways & Limitations

    The pre-trained representation adapts across diverse V+L tasks, including tasks very different from pre-training with minimal top-layer surgery.

  • Takeaways & Limitations

    NLVR2 uses two images as input, so directly fine-tuning an image-sentence-pair-pre-trained UNITER may not lead to optimal performance.

Abstract

from arXiv · show

Joint image-text embedding is the bedrock for most Vision-and-Language (V+L) tasks, where multimodality inputs are simultaneously processed for joint visual and textual understanding. In this paper, we introduce UNITER, a UNiversal Image-TExt Representation, learned through large-scale pre-training over four image-text datasets (COCO, Visual Genome, Conceptual Captions, and SBU Captions), which can power heterogeneous downstream V+L tasks with joint multimodal embeddings. We design four pre-training tasks: Masked Language Modeling (MLM), Masked Region Modeling (MRM, with three variants), Image-Text Matching (ITM), and Word-Region Alignment (WRA). Different from previous work that applies joint random masking to both modalities, we use conditional masking on pre-training tasks (i.e., masked language/region modeling is conditioned on full observation of image/text). In addition to ITM for global image-text alignment, we also propose WRA via the use of Optimal Transport (OT) to explicitly encourage fine-grained alignment between words and image regions during pre-training. Comprehensive analysis shows that both conditional masking and OT-based WRA contribute to better pre-training. We also conduct a thorough ablation study to find an optimal combination of pre-training tasks. Extensive experiments show that UNITER achieves new state of the art across six V+L tasks (over nine datasets), including Visual Question Answering, Image-Text Retrieval, Referring Expression Comprehension, Visual Commonsense Reasoning, Visual Entailment, and NLVR$^2$. Code is available at https://github.com/ChenRocks/UNITER.

1 Introduction

UNITER addresses the task-specific nature of existing multimodal representations by learning a universal image-text embedding through large-scale pre-training. Its design combines conditional masking and Optimal-Transport-based word-region alignment, and achieves strong performance across diverse V+L tasks.

  • Motivation: Existing V+L representations are often tailored to individual tasks, motivating a universal image-text representation.The paper frames generalization across heterogeneous V+L tasks as an open question.
  • Approach: UNITER uses a Transformer-based joint embedding model with image and text embedders, trained through four pre-training tasks.The tasks are MLM, MRM, ITM, and WRA.
  • Approach: Conditional masking observes the complete other modality during masked language or region modeling instead of masking both modalities jointly.This design is presented as a difference from previous multimodal pre-training approaches.
  • Approach: Optimal Transport-based WRA explicitly encourages fine-grained alignment between words and image regions, complementing global image-text matching.The paper reports that conditional masking and OT-based WRA ease image-text misalignment and improve joint embeddings.
  • Evaluation: UNITER is evaluated on six V+L tasks across nine datasets using COCO, Visual Genome, Conceptual Captions, and SBU Captions for pre-training.The evaluation includes VQA, VCR, NLVR2, Visual Entailment, Image-Text Retrieval, and Referring Expression Comprehension.
  • Results: UNITER achieves new state-of-the-art results across the reported V+L benchmarks, with extensive analysis of pre-training tasks and datasets.The contribution statement emphasizes broad benchmark gains and ablation-based analysis.

2 Related Work

Prior multimodal pre-training methods learn joint representations from paired visual and textual data, using either separate streams or a shared stream. UNITER distinguishes itself through conditional masking, explicit word-region alignment, and broader ablation analysis.

  • Multimodal Pre-training: Multimodal self-supervised methods pre-train on large-scale image or video and text pairs before downstream fine-tuning.Examples include VideoBERT, CBT, ViLBERT, LXMERT, B2T2, VisualBERT, Unicoder-VL, VL-BERT, and VLP.
  • Architecture Families: Two-stream architectures process images and text with separate Transformers before fusing them with a later Transformer.ViLBERT and LXMERT are cited as examples.
  • Architecture Families: Single-stream architectures apply one Transformer jointly to visual and textual inputs.B2T2, VisualBERT, Unicoder-VL, and VL-BERT are cited as examples.
  • UNITER’s Differences: UNITER uses conditional masking and introduces Optimal-Transport-based Word-Region Alignment, whereas prior alignment was mainly implicit in task-specific losses.The paper also studies pre-training-task combinations through extensive ablation experiments.

3 UNiversal Image-TExt Representation

UNITER embeds image regions and textual tokens in a shared space, then applies a multi-layer Transformer for cross-modal contextualization. Its pre-training combines conditional MLM and MRM with image-text matching and Optimal-Transport-based word-region alignment.

  • Model Architecture: UNITER maps visual regions and textual tokens into a common embedding space before applying a multi-layer Transformer.Image and text embedders provide modality-specific inputs for cross-modality contextualized embeddings.
  • Model Architecture: The image embedder combines Faster R-CNN visual features with seven-dimensional region-location features.The combined features are projected into the shared embedding space and normalized.
  • Model Architecture: The text embedder tokenizes sentences into WordPieces and sums word and position embeddings before layer normalization.A modality embedding is also used to distinguish visual and textual inputs.
  • Masked Modeling: MLM randomly masks 15% of input words and predicts them from the remaining words and all image regions.The masked tokens are replaced with [MASK] following the stated masking procedure.
  • Alignment Tasks: ITM learns instance-level image-sentence alignment by classifying sampled image-text pairs as matching or non-matching.The [CLS] representation is used as the fused joint representation for prediction.
  • Alignment Tasks: WRA learns a transport plan between word and region embeddings, using OT distance as an alignment loss.The transport plan represents cross-modal alignment, and IPOT approximates the computationally intractable exact optimization.
  • Masked Modeling: MRM masks image regions with probability 15% and reconstructs them from the remaining regions and all words.Because visual features are continuous, MRM uses feature regression or classification-based variants.
  • Masked Region Modeling Variants: MRC uses detector-derived hard object labels, while MRC-kl uses the detector’s soft class distribution and minimizes KL divergence.MRC-kl avoids treating the detector’s most likely class as guaranteed ground truth.

4 Experiments

UNITER is evaluated across six heterogeneous V+L tasks using standard benchmark metrics and ablation analyses. Results show that conditional masking, combined pre-training objectives, and suitable dataset composition improve performance, with strong gains across downstream tasks.

  • Experimental setup: UNITER is evaluated on six V+L tasks, using task-specific metrics and end-to-end fine-tuning of pre-trained models.The experiments use UNITER-base and UNITER-large; the benchmark suite includes VQA, VCR, NLVR2, Visual Entailment, Image-Text Retrieval, and Referring Expression Comprehension.
  • Ablation studies: +30 Meta-Sum over the no-pretraining baseline is obtained by text-only MLM initialization.Although text-only MLM does not use image information during pre-training, its weights initialize subsequent experiments.
  • Ablation studies: MLM + ITM improves over either single-task ITM or MLM, while joint MLM, ITM, and MRM training yields further gains.The ablation compares pre-training configurations across VQA, NLVR2, Flickr30K retrieval, and RefCOCO+ using Meta-Sum as a global metric.
  • Dataset effects: 405.24 Meta-Sum is achieved when in-domain and out-of-domain data are combined, exceeding 400.93 from in-domain data alone and 396.91 from out-of-domain data.The results indicate that similar downstream images remain beneficial, while additional out-of-domain data further improves performance when combined with in-domain data.
  • Downstream results: UNITER-large achieves new state of the art across all benchmarks, while UNITER-base posts gains over SOTA on VCR, NLVR2, SNLI-VE, retrieval, and referring expression comprehension.Reported UNITER-base improvements are approximately +2.8% on VCR Q→AR, +2.5% on NLVR2, +7% on SNLI-VE, +4% retrieval R@1, and +2% referring expression comprehension.
  • Downstream results: A second VCR pre-training stage significantly boosts performance, while first-stage generic pre-training provides more limited effects.The second stage uses VCR data because its images and text differ substantially from the standard pre-training datasets.
  • Downstream results: Pair-biattn achieves the best NLVR2 performance by adding bidirectional attention between paired-image representations.This modification compensates for the lack of cross-attention between images and adapts UNITER to the two-image NLVR2 input.

5 Conclusion

UNITER provides universal image-text representations through large-scale pre-training and extensive task ablations. It outperforms state-of-the-art models across multiple Vision-and-Language tasks, while motivating future work on pixel-token interaction and improved pre-training tasks.

  • UNITER provides universal image-text representations for Vision-and-Language tasks through large-scale pre-training.
  • Four main pre-training tasks are evaluated through extensive ablation studies.
  • UNITER outperforms state-of-the-art models over multiple Vision-and-Language tasks by a significant margin.
  • Future work includes early interaction between raw image pixels and sentence tokens and more effective pre-training tasks.

A Appendix

The supplementary material contains eight sections covering dataset collection, downstream implementation, masking comparisons, additional results, model comparisons, and optimal transport background.

  • The supplementary material has eight sections covering dataset collection and downstream-task implementation details.
  • It includes quantitative comparisons between conditional masking and joint random masking.
  • Additional sections report results on VCR and NLVR2 and compare UNITER directly with VLBERT and ViLBERT.
  • The supplement also provides background on optimal transport and the IPOT algorithm.

A.1 Dataset Collection

UNITER’s pre-training dataset combines four existing Vision-and-Language datasets while removing downstream evaluation images to prevent pre-training exposure. The collection also excludes co-occurring Flickr30K images to preserve fair zero-shot retrieval evaluation.

  • Dataset Composition: UNITER’s full dataset comprises COCO, Visual Genome, Conceptual Captions, and SBU Captions.
  • Evaluation Separation: Downstream evaluation images are excluded because validation and test images are scattered across raw COCO splits.
  • Evaluation Separation: Co-occurring Flickr30K images are removed through URL matching to keep zero-shot image-text retrieval evaluation fair.
  • Evaluation Separation: Figure 4 lists downstream-task splits from VQA, Image-Text Retrieval, COCO Captioning, RefCOCO variants, and BUTD detection.

A.2 Implementation Details

The appendix details UNITER’s hardware, optimization, and task-specific fine-tuning procedures. It also documents additional pre-training for VCR, architecture variants for NLVR2, retrieval hard-negative sampling, and a detector-feature fairness caveat.

  • Infrastructure: Pre-training uses Nvidia V100 GPUs with mixed precision, Horovod, NCCL, dynamic sequence lengths, and gradient accumulation.
  • Downstream Tasks: VQA fine-tuning uses a 3129-answer multi-label classifier trained with binary cross-entropy.
  • Downstream Tasks: VCR is trained on question-answering and answer-justification subtasks, with cross-entropy classification over right-or-wrong choices.
  • Downstream Tasks: For VCR data unlike pre-training data, second-stage pre-training uses MLM, MRFR, and MRC-kl while discarding ITM.
  • Downstream Tasks: NLVR2 supports Triplet, Pair, and Pair-biattn variants that combine image-text representations for binary classification.
  • Downstream Tasks: Retrieval fine-tuning adds hard negatives by selecting top-ranked negative sentences and images from sparse scoring matrices.

A.3 Conditional Masking vs. Joint Random Masking

Conditional masking uses the fully observed opposite modality to learn cross-modal context, and it outperforms joint random masking during pre-training. Validation curves show faster convergence and higher final accuracy for both masked words and masked regions.

  • A.3 Conditional Masking vs. Joint Random Masking: Conditional masking infers masked regions from surrounding regions and the full sentence, rather than jointly masking both modalities.The same principle applies to masked language modeling, which uses the full image as context.
  • A.3 Conditional Masking vs. Joint Random Masking: MLM accuracy measures masked-word reconstruction, while MRC-kl accuracy measures correct classification of masked regions.For MRC-kl validation, correctness is determined by selecting the most confident predicted category.
  • A.3 Conditional Masking vs. Joint Random Masking: Conditional masking converges faster and achieves higher final accuracy than joint random masking for both MLM and MRC-kl.Figure 6 compares the two masking strategies through validation curves for the two tasks.

A.4 More Ablation Studies on Pre-training Settings

Additional ablations examine MRC-only pre-training and WRA. MRC-only is a weak setting, whereas WRA provides task-dependent benefits, especially for region-level recognition and reasoning.

  • A.4 More Ablation Studies on Pre-training Settings: MRC-only pre-training performs similarly to MRFR-only pre-training and is weaker than the other in-domain pre-training settings.These results are reported for UNITER-base on in-domain data.
  • A.4 More Ablation Studies on Pre-training Settings: WRA significantly improves VQA and RefCOCO+ performance while producing comparable results on Flickr and NLVR2.The WRA ablation uses UNITER-large with in-domain and out-of-domain data; Table 8 reports R@1 for zero-shot retrieval settings.
  • A.4 More Ablation Studies on Pre-training Settings: WRA mainly benefits tasks relying on region-level recognition and reasoning because it encourages local alignment between image regions and words.Flickr and NLVR2 focus more on global rather than local alignments.
  • A.4 More Ablation Studies on Pre-training Settings: Adding WRA yields large zero-shot image-text retrieval gains and consistent gains across the other evaluated tasks.This finding comes from additional UNITER-large ablations using both in-domain and out-of-domain data.

A.5 More Results on VCR and NLVR2

Further evaluations report strong UNITER results on VCR and NLVR2. UNITER-large surpasses prior systems on an additional NLVR2 test split, while an ensemble and even a single model outperform competing VCR systems.

  • A.5 More Results on VCR and NLVR2: 7.0%: a 10-model UNITER-large ensemble surpasses the ViLBERT ensemble on VCR Q→AR accuracy.A single UNITER-large also outperforms the ViLBERT ensemble and VLBERT-large by 3.0%.
  • A.5 More Results on VCR and NLVR2: 3.0%: a single UNITER-large outperforms the ViLBERT ensemble and VLBERT-large on VCR Q→AR accuracy.The comparison follows the VCR setup used in Table 4 of the main paper.
  • A.5 More Results on VCR and NLVR2: ∼4.0%: UNITER-large outperforms the previous state of the art on all metrics of the additional NLVR2 test split.The comparison is against LXMERT and VisualBERT.
  • A.5 More Results on VCR and NLVR2: The NLVR2 additional split reports balanced and unbalanced evaluations, which were introduced in the cited prior work.The passage identifies these as the evaluation types for the reported metrics.

A.6 Direct Comparison to VLBERT and ViLBERT

A direct Conceptual Captions-only comparison shows that UNITER retains an advantage over ViLBERT and VLBERT on VQA and RefCOCO+. The section also reviews optimal transport and the IPOT approximation used to compute alignment.

  • A.6 Direct Comparison to VLBERT and ViLBERT: UNITER consistently outperforms ViLBERT and VLBERT on VQA and RefCOCO+ when all models are trained only on Conceptual Captions.UNITER uses conditional masking and its best pre-training task configuration in this comparison.
  • A.7 Review of Optimal Transport and the IPOT Algorithm: Optimal transport defines a distance between probability measures by minimizing the transport cost between them.In UNITER, the same framework supports alignment between contextualized image-region and word embeddings.
  • A.7 Review of Optimal Transport and the IPOT Algorithm: IPOT approximates the optimal transport distance with an efficient iterative scheme because exact minimization over the transport matrix is generally computationally intractable.The method uses a proximal-point formulation and returns an approximate OT matrix and distance.
  • A.7 Review of Optimal Transport and the IPOT Algorithm: Sinkhorn is not used for model training because its numerical stability and performance are sensitive to the regularization hyper-parameter ϵ.The experiments therefore use IPOT instead.
Loading 1909.11740v3…