PaperlayerView exact revision on arXiv

Source-linked AI summary

Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training

Gen Li, Nan Duan, Yuejian Fang, Ming Gong, Daxin Jiang, Ming Zhou

arXiv:1908.06066v3cs.CV

TL;DR

Cross-modal tasks need models that jointly handle long language sequences and visual content, which separate vision and language pre-training does not provide. Unicoder-VL addresses this with a Transformer trained on image-caption pairs using MLM, MOC, and VLM, and transfers it to retrieval and VCR with strong results. The paper reports state-of-the-art or comparable performance across these tasks, while leaving image-only pre-training as future work.

Abstract

from arXiv · show

We propose Unicoder-VL, a universal encoder that aims to learn joint representations of vision and language in a pre-training manner. Borrow ideas from cross-lingual pre-trained models, such as XLM and Unicoder, both visual and linguistic contents are fed into a multi-layer Transformer for the cross-modal pre-training, where three pre-trained tasks are employed, including Masked Language Modeling (MLM), Masked Object Classification (MOC) and Visual-linguistic Matching (VLM). The first two tasks learn context-aware representations for input tokens based on linguistic and visual contents jointly. The last task tries to predict whether an image and a text describe each other. After pretraining on large-scale image-caption pairs, we transfer Unicoder-VL to caption-based image-text retrieval and visual commonsense reasoning, with just one additional output layer. We achieve state-of-the-art or comparable results on both two tasks and show the powerful ability of the cross-modal pre-training.

Introduction

Existing vision and language pre-trained models handle their own modalities well but do not directly model long-sequence vision-language interactions. Unicoder-VL addresses this with joint cross-modal pre-training and achieves strong transfer results.

Motivation

  • Vision pre-trained models trained on image labels cannot directly handle cross-modal tasks with long natural-language inputs.Such tasks therefore require additional fusion layers to model visual-linguistic interactions.Source: Introduction · p. 1
  • Language pre-trained models handle long natural-language sequences but are not trained directly with visual content.They learn contextualized text representations from large-scale corpora before downstream fine-tuning.Source: Introduction · p. 1

Unicoder-VL

  • Unicoder-VL uses a multi-layer Transformer to learn joint vision-language representations through cross-modal pre-training on large-scale image-caption pairs.The framework is inspired by BERT and cross-lingual pre-trained models such as XLM and Unicoder.Source: Introduction · p. 1
  • Three tasks are used: Masked Language Modeling, Masked Object Classification, and Visual-linguistic Matching.MLM and MOC learn context-aware token representations jointly, while VLM predicts whether an image and text describe each other.Source: Introduction · p. 1

Results

  • State-of-the-art results were achieved on MSCOCO and Flickr30K retrieval, with comparable results on visual commonsense reasoning.The model also showed good zero-shot performance, indicating generalization ability.Source: Introduction · pp. 1–2

Related Work

Prior vision and language pre-training approaches differ in architecture, masking objectives, and pre-training data. Unicoder-VL uses a single-stream cross-modal design with direct visual features and attains strong retrieval and VCR results.

Pre-training for CV Tasks

  • ImageNet-based vision models are poorly suited to cross-modal tasks requiring long natural-language inputs because they are trained only on image labels.Unicoder-VL instead uses image-caption pairs, making it more suitable for these tasks.Source: Pre-training for CV Tasks · p. 2

Pre-training for NLP Tasks

  • NLP pre-trained models learn contextualized text representations from large-scale corpora and adapt to downstream tasks through fine-tuning.Their pre-training does not directly include visual content.Source: Pre-training for NLP Tasks · p. 2

Cross-modal Pre-training

  • VideoBERT combines transcript words with generated visual words, whereas Unicoder-VL jointly trains on visual object features and linguistic content.This distinguishes direct visual-feature modeling from discretized visual-word representations.Source: Pre-training for Cross-modal Tasks · p. 2

Architectures

  • ViLBERT uses separate vision and language networks followed by cross-modal Transformers, while Unicoder-VL belongs to the single-stream architecture group.VisualBERT, VL-BERT, and UNITER are also described as single-stream models.Source: Pre-training for Cross-modal Tasks · p. 3

Pre-training Tasks

  • The compared models vary in object prediction, masking strategy, and use of visual-linguistic matching during pre-training.All listed models use masked language modeling, but their treatment of object prediction and modality masking differs.Source: Pre-training for Cross-modal Tasks · p. 3

Results

  • Unicoder-VL achieves state-of-the-art results on image-to-text retrieval, text-to-image retrieval, and VCR compared with recent works.The paper presents this as evidence of the model’s ability on these tasks.Source: Pre-training for Cross-modal Tasks · p. 3

Approach

Unicoder-VL encodes image regions and text tokens together with a self-attention Transformer, then pre-trains the joint representation using three cross-modal objectives. Visual inputs come from fixed Faster R-CNN region features.

Model Background

  • BERT provides the multi-layer Transformer foundation and uses masked language modeling and next sentence prediction during pre-training.Unicoder-VL extends this foundation to visual and linguistic inputs.Source: BERT · p. 3, Approach · p. 3

Unicoder-VL Encoder

  • Image regions and textual tokens are encoded and passed through multi-layer self-attention to learn cross-modality contextualized embeddings.The resulting embeddings jointly represent visual regions and textual tokens.Source: Unicoder-VL · p. 3

Input Embeddings

  • Text inputs use WordPiece tokens with [CLS], [SEP], and [IMG] special tokens, while linguistic representations combine word and position embeddings.The embeddings are initialized from BERT.Source: Unicoder-VL · p. 3
  • Image inputs use Faster R-CNN to extract pooled region features and location features, which are projected into a shared embedding space.The final region representation sums the projected visual and location features before layer normalization.Source: Unicoder-VL · p. 3
  • The Faster R-CNN model remains fixed during Unicoder-VL training.Its predicted object labels are retained for the object-label prediction task.Source: Unicoder-VL · p. 3

Pre-training Tasks

  • The three pre-training tasks are Masked Language Modeling, Masked Object Classification, and Visual-linguistic Matching.Together they provide language masking, object prediction, and image-sentence alignment objectives.Source: Unicoder-VL · pp. 3–4
  • MLM masks words with probability 15% and predicts them from surrounding words and all image regions.The objective is a negative log-likelihood over masked words.Source: Unicoder-VL · p. 3
  • MOC masks sampled visual features and predicts their object categories from the Transformer output.The target category is the highest-confidence label predicted by the detection model.Source: Unicoder-VL · pp. 3–4

Experiments

The experiments pretrain and transfer Unicoder-VL across image-text retrieval and visual commonsense reasoning, using large-scale image-caption data and task-specific fine-tuning procedures.

Pretraining setup

  • 3.8M image-caption pairs are used for pretraining, combining approximately 3M Conceptual Captions pairs with 0.8M SBU pairs.The model uses 12 Transformer layers, 768 hidden units, 12 attention heads, and a maximum sequence length of 144.Source: Pre-training Unicoder-VL · p. 4

Image-Text Retrieval

  • Retrieval evaluation uses MSCOCO and Flickr30K, with R@1, R@5, and R@10 measuring the percentage of ground-truth matches in the top K results.MSCOCO and Flickr30K provide separate training, validation, and testing splits for evaluating both retrieval directions.Source: Image-Text Retrieval · p. 4
  • Image-text retrieval is formulated as ranking with image-to-text and text-to-image matching tasks, sampling three negative cases per matching task.Triplet loss maximizes the margin between positive and negative pairs after computing cross-modal similarity scores.Source: Image-Text Retrieval · p. 4
  • The zero-shot retrieval setting applies the pretrained visual-linguistic alignment mechanism directly without task-specific fine-tuning.This setting tests whether pretraining can ground text and generalize across visual and linguistic variation.Source: Zero-shot Image-Text Retrieval · p. 5

Visual Commonsense Reasoning

  • VCR fine-tuning concatenates each question with candidate answers or rationales, passes four image-text inputs through the model, and predicts with a softmax over four scores.Ground-truth object boxes are matched to extracted visual features, and referenced visual features are additionally attached to corresponding language tokens.Source: Visual Commonsense Reasoning · p. 5

Results and Analysis

Unicoder-VL’s pre-train-plus-fine-tune setting outperformed retrieval baselines across evaluation metrics, with larger gains on Flickr30k, while VCR improvements were comparatively modest. Ablations indicate benefits from larger Transformer encoders and more pre-training data, alongside limitations from domain mismatch and unfine-tuned detection backbones.

Image-Text Retrieval

  • Unicoder-VL’s pre-train-plus-fine-tune setting significantly outperformed all baselines on every image-text retrieval evaluation metric.The comparison covers zero-shot, task-specific training, and pre-train-plus-fine-tune settings.Source: Results on Image-Text Retrieval · pp. 5–6
  • 7.8% and 8.1% absolute R@1 improvements over PFAN were obtained on the MSCOCO 1K sentence-retrieval and image-retrieval tasks, respectively.On Flickr30k, Unicoder-VL reached 86.2% and 71.5% R@1 for sentence and image retrieval, with larger gains over PFAN.Source: Results on Image-Text Retrieval · p. 6
  • 10.1 points of improvement over ViLBERT were reported, while Unicoder-VL achieved comparable results to UNITER despite using 1.8M fewer image-caption pairs.The authors note that UNITER’s additional data included in-domain Visual Genome captions.Source: Results on Image-Text Retrieval · p. 6

Visual Commonsense Reasoning

  • Unicoder-VL only slightly improved VCR performance with pre-training, although it outperformed R2C by large margins using only a simple classification layer.The authors attribute the limited pre-training gain to a gap between perceptual image-caption data and cognitively oriented VCR data.Source: Results on Visual Commonsense… · p. 6

Discussion

  • The method does not use image-only inputs because training failed to converge, and its Faster-RCNN backbone was not fine-tuned jointly with cross-modal training.The authors leave extension to image-only tasks and detection-backbone fine-tuning for future work.Source: Discussion · p. 6

Ablation Studies

Conclusion

Unicoder-VL uses large-scale image-caption pretraining and three cross-modal tasks to learn aligned visual-linguistic representations. Fine-tuning demonstrates strong retrieval gains, zero-shot transfer, and improved visual commonsense reasoning.

  • Unicoder-VL pretrains on large-scale image-caption pairs with three tasks that align visual and linguistic modalities.The tasks are introduced to learn better cross-modal representations.Source: Conclusion · p. 7
  • Unicoder-VL significantly boosts image and sentence retrieval performance on Flickr30K and MSCOCO after fine-tuning.Source: Conclusion · p. 7
  • Zero-shot experiments show that Unicoder-VL applies learned cross-modal knowledge directly to image and sentence retrieval without task-specific fine-tuning.Source: Conclusion · p. 7
  • Cross-modal pretraining improves visual commonsense reasoning, although the method is presented as broadly applicable beyond the evaluated tasks.Source: Conclusion · p. 7

Related papers

Loading 1908.06066v3…