PaperlayerView exact revision on arXiv

Source-linked AI summary

UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning

Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, Haifeng Wang

arXiv:2012.15409v4cs.CL

TL;DR

Prior pre-training methods are divided between single-modal learning and limited-pair multi-modal learning, restricting adaptation across settings. UNIMO learns from text, images, and image-text pairs, aligning them with CMCL in a unified semantic space. The authors report improved performance across single-modal and multi-modal downstream understanding and generation tasks.

  • Problem

    Existing pre-training methods focus on either single-modal tasks or limited image-text-pair learning, making adaptation to the other setting difficult.

  • Method

    UNIMO jointly learns visual and textual representations from image collections, text corpora, and image-text pairs, aligning them through cross-modal contrastive learning.

  • Results

    UNIMO outperforms previous methods on both multi-modal and single-modal downstream tasks, including understanding and generation.

  • Takeaways & Limitations

    A unified semantic space allows textual and visual knowledge to mutually enhance each other across downstream modality settings.

Abstract

from arXiv · show

Existed pre-training methods either focus on single-modal tasks or multi-modal tasks, and cannot effectively adapt to each other. They can only utilize single-modal data (i.e. text or image) or limited multi-modal data (i.e. image-text pairs). In this work, we propose a unified-modal pre-training architecture, namely UNIMO, which can effectively adapt to both single-modal and multi-modal understanding and generation tasks. Large scale of free text corpus and image collections can be utilized to improve the capability of visual and textual understanding, and cross-modal contrastive learning (CMCL) is leveraged to align the textual and visual information into a unified semantic space over a corpus of image-text pairs. As the non-paired single-modal data is very rich, our model can utilize much larger scale of data to learn more generalizable representations. Moreover, the textual knowledge and visual knowledge can enhance each other in the unified semantic space. The experimental results show that UNIMO significantly improves the performance of several single-modal and multi-modal downstream tasks. Our code and pre-trained models are public at the UNIMO project page https://unimo-ptm.github.io/

1 Introduction

UNIMO addresses the gap between single-modal and multi-modal pre-training by learning from text, images, and image-text pairs in one unified architecture. It aligns visual and textual representations through CMCL, enabling adaptation to understanding and generation tasks across modalities.

  • Motivation: Existing single-modal methods do not effectively support multi-modal scenarios, while cross-modal methods rely on limited image-text pairs and generalize poorly to single-modal tasks.The paper identifies modality-specific pre-training and limited paired data as central limitations of prior approaches.
  • UNIMO: UNIMO is designed to process textual, visual, and vision-and-language inputs with one unified-modal architecture.The architecture is motivated by the complementary information available across modalities.
  • UNIMO: Cross-modal contrastive learning aligns visual and textual representations into a shared semantic space using image-text pairs.Text rewriting provides diverse positive and hard-negative cross-modal examples for different levels of semantic alignment.
  • UNIMO: UNIMO uses large-scale non-paired text corpora and image collections to learn more generalizable textual and visual representations.This expands training beyond image-text pairs and supports both vision and language understanding and generation.
  • Advantages: The model can be fine-tuned for both single-modal and multi-modal understanding and generation downstream tasks.This capability is presented as a central advantage over methods specialized for only one task or modality setting.
  • Advantages: Visual and textual knowledge can enhance one another, improving performance on several single-modal and multi-modal tasks.The paper attributes this benefit to learning both modalities in a unified semantic space.

2 UNIMO

UNIMO unifies visual and textual representations in a shared semantic space while learning from images, texts, and image-text pairs. Its CMCL uses multi-granularity rewritten pairs, retrieved single-modal samples, and contrastive learning to align modalities for broader understanding and generation.

  • 2 UNIMO: UNIMO learns modality-invariant representations from images, texts, and image-text pairs using a unified-modal Transformer.The model extracts semantic representations for images and texts and jointly learns them across multiple data types.
  • 2.1 Cross-Modal Contrastive Learning: Existing cross-modal methods rely on limited image-text pairs and coarse matching, which restricts generalization to single-modal scenarios.Randomly sampled negatives are often dissimilar to the original pair, producing only coarse visual-textual alignment.
  • 2.1 Cross-Modal Contrastive Learning: CMCL brings paired images and texts closer while separating non-paired examples, aligning visual and textual representations at multiple semantic levels.Text rewriting creates word-, phrase-, and sentence-level positive and hard-negative examples for contrastive learning.
  • 2.1 Cross-Modal Contrastive Learning: Retrieved related images and texts from single-modal collections provide additional positive samples during cross-modal learning.Retrieved images are selected by visual similarity and object overlap, while retrieved sentences are selected by semantic similarity.
  • 2.2 Visual Learning: Visual learning masks image regions and combines feature regression with region classification to reconstruct masked visual information.For image-text pairs, masked regions are reconstructed using both the remaining regions and the accompanying text.
  • 2.3 Language Learning: Seq2Seq generation samples text fragments as targets and trains the model to generate them autoregressively from the remaining source sequence.This objective is applied to captions in image-text pairs to support cross-modal understanding and generation.

3 Experimental Settings

UNIMO is evaluated using three pre-training data types and two model sizes across single-modal and multimodal downstream tasks. The setup combines text, image, and image-text data with contrastive and visual-region processing.

  • 3.1 Pre-training Dataset: UNIMO is pre-trained on text corpora, image collections, and image-text pairs.The text data include BookWiki and OpenWebText; images include OpenImages and COCO unlabeled data; paired data come from COCO, Visual Genome, Conceptual Captions, and SBU Captions.
  • 3.2 Pre-training Settings: UNIMO-base uses 12 Transformer layers, while UNIMO-large uses 24 Transformer layers.Both models are initialized from corresponding RoBERTa checkpoints and trained for at least 500K steps.
  • 3.2 Pre-training Settings: Visual learning uses Faster R-CNN region features, retaining up to 100 image boxes whose detection probability exceeds 0.2.For CMCL, the setup creates 3 positive samples and 100 hard negative samples per image-text pair.
  • 3.3 Finetuning Tasks: The evaluation covers single-modal language understanding and generation alongside multimodal vision-language understanding and generation.Single-modal tasks include conversational question answering, question generation, summarization, sentence compression, and understanding benchmarks; multimodal tasks are also evaluated.

4 Results and Analysis

UNIMO performs strongly across multimodal and single-modal tasks, and ablations show that textual and visual knowledge mutually support performance. Removing single-modal learning or either modality reduces performance in the corresponding settings.

  • 4.1 Multi-Modal tasks: UNIMO achieves the best results on almost all evaluated multimodal benchmarks for both base and large models.UNIMO-large exceeds ERNIE-ViL-large by 1.34 R@1 on image retrieval and 1.3 R@1 on text retrieval, and beats Oscar by more than 2 BLUE4 on image captioning.
  • 4.2 Single-Modal tasks: Removing single-modal learning causes dramatic drops across all evaluated language understanding and generation tasks.The ablated model replaces CMCL with image-text matching and resembles a conventional multimodal pre-training method.
  • 4.2 Single-Modal tasks: UNIMO performs better or comparably to established language pre-trained models on language understanding and generation tasks.It outperforms UniLM on most tasks with a large margin.
  • 4 Results and Analysis: Overall, UNIMO performs well on both multimodal and single-modal tasks, supporting the adaptability of its unified-modal architecture.The experiments also examine whether textual and visual knowledge mutually enhance one another in a unified semantic space.
  • 4.2 Single-Modal tasks: Removing textual learning consistently lowers performance on both multimodal understanding and generation tasks.The ablation supports a benefit from textual knowledge for cross-modal learning.
  • 4.2 Single-Modal tasks: Removing visual data reduces performance on single-modal language tasks.Without images and image-text pairs, the model is trained only with language objectives similar to BERT and UniLM.

5 Related Work

Prior work separates single-modal and multimodal pre-training, with multimodal methods typically relying on image-text pairs. UNIMO is presented as a unified alternative that uses non-paired text and images while supporting both task types.

  • Single-Modal Pre-training: Single-modal pre-training methods focus on either visual or language tasks.Visual approaches include CNN-based ImageNet pre-training and contrastive self-supervised learning; language approaches include established Transformer language models.
  • Multi-Modal Pre-training: Multimodal pre-training methods generally learn from image-text pairs using BERT-like objectives.Representative systems include ViLBERT, VisualBERT, VL-BERT, Unicoder-VL, and UNITER.
  • Multi-Modal Pre-training: Contrastive multimodal methods such as CLIP and WenLan use large-scale image-text pairs to learn transferable cross-modal representations.These approaches emphasize image-text matching and contrastive cross-modal learning.
  • Multi-Modal Pre-training: UNIMO differs by using large volumes of non-paired text and images alongside paired data and adapting to textual and multimodal scenarios.The paper reports that UNIMO achieves the best performance across the evaluated settings.

6 Conclusion

UNIMO unifies non-paired text and image data for cross-modal learning and aligns their knowledge in a shared semantic space. It adapts to both single-modal and multimodal understanding and generation tasks, outperforming prior methods in the reported evaluations.

  • 6 Conclusion: UNIMO leverages large-scale non-paired text corpora and image collections for cross-modal learning.Its unified semantic space enables textual and visual knowledge to mutually enhance one another.
  • 6 Conclusion: UNIMO successfully adapts to both single-modal and multimodal understanding and generation tasks.The conclusion reports improved performance over previous methods on both categories of downstream tasks.

A Pre-training Settings

UNIMO pre-training represents images, texts, and image-text pairs in a shared format, jointly optimizing visual, language, and CMCL objectives. CMCL uses positive and negative cross-modal examples to align representations in a unified semantic space.

  • Data representation: Images, texts, and image-text pairs are represented in a common format containing visual regions and textual tokens.Single-modal inputs use masked pseudo-sequences so each modality can be encoded individually while sharing the image-text representation format.
  • Data mixture: The pre-training data mix combines image collections, text corpus, and image-text pairs with sampling ratio 1:1:5.
  • Cross-modal contrastive learning: CMCL maximizes scores for paired and related image-text examples while minimizing scores for negative pairs, images, and texts.The loss includes positive paired, related-image, and related-text terms alongside corresponding negative terms.
  • Data augmentation: Text rewriting and image/text retrieval augment image-text pairs with positive and hard-negative examples for multi-granularity semantic alignment.The augmentations include back-translated captions and related items retrieved from single-modal collections and corpora.
  • Dataset statistics: The pre-training datasets include image-text pairs, image collections, and text corpus, with their statistics reported in Table 5.

B Finetuning Settings

UNIMO is evaluated after fine-tuning on multimodal understanding, generation, entailment, and retrieval tasks, using task-specific datasets and reported hyper-parameters. Its pre-training algorithm jointly processes mixed unimodal data and augmented image-text examples.

  • Multimodal tasks: UNIMO fine-tuning covers VQA, image captioning, visual entailment, and image-text retrieval tasks.VQA answers questions from images, captioning generates image descriptions, entailment predicts logical relationships, and retrieval evaluates both target modalities.
  • Pre-training procedure: The pre-training algorithm randomly mixes image samples, text samples, and image-text pairs, then jointly optimizes visual, language, and CMCL losses.For each image-text pair, the loader adds rewritten positives and negatives plus retrieved positive images and texts.
  • Experimental configuration: Dataset statistics and fine-tuning hyper-parameters are reported for the multimodal and single-modal downstream tasks.
  • Evaluation metrics: Image-text retrieval reports R@1, R@5, and R@10 on Flickr30k image-retrieval and text-retrieval subtasks.R denotes Recall in the reported top-K retrieval results.

C Visualization and Analysis

Visualization and retrieval examples provide qualitative evidence that unified-modal learning captures both visual and semantic relationships. UNIMO retrieves more detailed cross-modal matches and identifies background concepts beyond purely visual neighbors.

  • Embedding visualization: PCA visualization compares UNIMO with ablations by displaying nearest-neighbor words around center-word embeddings.The analysis uses a two-dimensional embedding visualization to inspect unified-modal learning.
  • Ablation comparison: “UNIMO-w/o texts” finds more visually relevant words than “UNIMO-w/o image&pairs,” demonstrating the contribution of visual learning on images.
  • Embedding analysis: UNIMO finds visually relevant words and semantic background concepts such as “lunch” and “airplanes” for “hamburger.”For “foot,” it also identifies “meter,” “steps,” and “soccer,” enriching the associated concept.
  • Text retrieval: UNIMO’s text-retrieval examples describe images more accurately by including objects, attributes, and relations.Figure 5 marks accurate visual information in green and wrong information in red.
  • Image retrieval: UNIMO’s image-retrieval examples better match captions with more detailed semantic alignments.Figure 6 uses blue to mark important information neglected by the baseline but recognized by UNIMO.
Loading 2012.15409v4…