PaperlayerView exact revision on arXiv

Source-linked AI summary

UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training

Mingyang Zhou, Luowei Zhou, Shuohang Wang, Yu Cheng, Linjie Li, Zhou Yu, Jingjing Liu

arXiv:2104.00332v1cs.CV

TL;DR

Cross-lingual multimodal models have faced limited training corpora and difficulty aligning vision directly with non-English languages. UC2 augments English-only image-caption data with machine translation, uses images as a shared pivot, and adds two pre-training tasks. It reports state-of-the-art performance on multiple multilingual retrieval and VQA benchmarks.

  • Problem

    Cross-lingual multimodal models have been validated on small task-specific datasets, and English-anchored training can make direct vision-to-non-English alignment difficult.

    Source: §1 · p. 1

  • Method

    UC2 machine-translates English-only image-caption datasets and trains multilingual objectives with images as the primary pivot, adding MRTM and VTLM.

    Source: §1 · p. 2, §3.3.1 · p. 4

  • Results

    UC2 achieves new state of the art on multiple multilingual image-text retrieval and VQA benchmarks.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    UC2 learns cross-lingual cross-modal representations from multilingual image-to-text pairs and demonstrates cross-lingual transfer capability.

    Source: §5 · p. 8

Abstract

from arXiv · show

Vision-and-language pre-training has achieved impressive success in learning multimodal representations between vision and language. To generalize this success to non-English languages, we introduce UC2, the first machine translation-augmented framework for cross-lingual cross-modal representation learning. To tackle the scarcity problem of multilingual captions for image datasets, we first augment existing English-only datasets with other languages via machine translation (MT). Then we extend the standard Masked Language Modeling and Image-Text Matching training objectives to multilingual setting, where alignment between different languages is captured through shared visual context (i.e, using image as pivot). To facilitate the learning of a joint embedding space of images and all languages of interest, we further propose two novel pre-training tasks, namely Masked Region-to-Token Modeling (MRTM) and Visual Translation Language Modeling (VTLM), leveraging MT-enhanced translated data. Evaluation on multilingual image-text retrieval and multilingual visual question answering benchmarks demonstrates that our proposed framework achieves new state-of-the-art on diverse non-English benchmarks while maintaining comparable performance to monolingual pre-trained models on English tasks.

1. Introduction

UC2 addresses the English bias and limited scale of cross-lingual multimodal learning by augmenting English-only image-caption data with machine translations and using images as the primary pivot. It combines multilingual versions of established objectives with two new tasks and reports state-of-the-art results across multilingual retrieval and VQA benchmarks.

  • Prior cross-lingual multimodal models were validated on small, task-specific datasets, making scaling and generalization to more languages non-trivial.Source: §1 · p. 1
  • M3P uses English as an anchor between images and other languages, which can introduce linguistic discrepancy for direct image-to-non-English alignment.Source: §1 · pp. 1–2
  • UC2 augments English-only image-caption datasets with machine-translated languages and pivots primarily on images, complementarily on English.The paper describes this as the first known effort to create large-scale training datasets with multilingual image captions.Source: §1 · p. 2
  • UC2 extends Masked Language Modeling and Image-Text Matching to multilingual training and introduces MRTM and VTLM as additional pre-training objectives.Source: §1 · p. 2
  • UC2 achieves new state of the art on multiple multilingual image-text retrieval and VQA benchmarks, including Multi30k and COCO.Source: §1 · p. 2

2. Related Work

Prior work spans vision-language pre-training, cross-lingual multimodal retrieval, and multimodal machine translation. UC2 is positioned as a unified pre-training approach that grounds sentences across languages in shared visual context.

  • Vision-language pre-training evolved from two-stream models that fuse visual and textual representations to single-stream models that encode concatenated inputs.Unified VLP extends to understanding and generation, while other work varies input features and uses multi-task learning.Source: §2 · p. 2
  • Cross-lingual multimodal studies align images and captions across languages, with approaches using English or images as pivots and others scaling through character-based or shared language-acoustic embeddings.SMALR learns visually aligned word embeddings to balance multilingual capacity and task performance.Source: §2 · p. 3
  • Multimodal machine translation incorporates visual information through attention, fusion, multi-task learning, Transformer models, or image-pivoted unsupervised translation.Source: §2 · p. 3
  • Earlier task-specific models used small datasets, while M3P alternates multimodal monolingual and mono-modal multilingual training but uses English as an anchor, making vision-to-non-English alignment difficult.UC2 instead grounds sentences in different languages on shared visual context.Source: §2 · p. 3

3. Cross-Lingual Cross-Modal Pre-training

UC2 builds multilingual cross-modal pre-training by augmenting English image-caption data with machine translations and training on multilingual image-text pairs. Its objectives use visual context to align languages and introduce region-to-token and bilingual translation modeling.

3.1. Machine Translation Augmented Dataset

  • 3.3 million images are paired with captions in six languages after Conceptual Captions are translated into German, French, Czech, Japanese, and Chinese.The translated captions are produced with Microsoft Azure Translation API Service.Source: §3.1 · p. 3

3.2. Model Overview

  • UC2 extends a monolingual V+L language encoder to a cross-lingual encoder, combining image and language features in a Transformer for contextualized cross-modal representations.Image regions come from Faster R-CNN; image-region and location features are projected into the text embedding dimension before being summed.Source: §3.2 · p. 3

3.3.1 General Tasks

  • MLM and ITM provide the multilingual base objectives: MLM predicts randomly masked caption words from surrounding words and image regions, while ITM distinguishes matched from mismatched image-text pairs.Words are masked with 15% probability; ITM uses equal-probability positive and negative pairs, with negatives formed using in-batch distractors.Source: §3.3.1 · p. 4

3.3.2 Masked Region-to-Token Modeling

  • MRTM predicts masked regions as detector-derived object-label tokens, connecting region semantics with caption vocabulary rather than predicting detector labels as unrelated indices.Early Adaptation projects image features into the word-embedding space and aligns their predicted distribution with the detector’s object probabilities; the paper contrasts this with OSCAR’s use of object tags at downstream time.Source: §3.3.2 · pp. 4–5

3.3.3 Visual Translation Language Modeling

  • VTLM jointly predicts masked tokens in an image and two captions from different languages, with one caption always English because translated captions are less reliable.The bilingual input grows linearly with the number of languages.Source: §3.3.3 · p. 5
  • Co-masking hides semantically aligned words in both captions to prevent cross-language translation alone from solving VTLM and encourage use of visual context.Fast Align supplies word alignments from the machine-translated parallel corpus.Source: §3.3.3 · p. 5

4. Experiments

Experiments evaluate UC2 on multilingual image-text retrieval and VQA, then test its pre-training objectives and image pivoting and visualize learned alignments.

4.1.1 Evaluation on Multilingual Retrieval

  • 17.3% Meta-Ave gain over M3P with English-only fine-tuning; all-language fine-tuning improves Meta-Ave 3.4% over M3P and beats UNITERCC except on MSCOCO English.With English-only fine-tuning, UC2 is also 5% better than the best non-pretrained models trained on data in each language.Source: Table 1 · p. 6, §4.1.1 · p. 7

4.1.2 Evaluation on Multilingual VQA

  • More than 10% improvement on both accuracy and BLEU over baselines on VG VQA Japanese, despite UC2 using 61K versus PCATT's 91K training images.On VQA v2.0, UC2 also improves over the state-of-the-art task-specific method and monolingual models pre-trained on Conceptual Conception.Source: Table 2 · p. 7, §4.1.2 · p. 7

4.1.3 Ablation Study

  • MRTM significantly boosts multilingual VQA and incrementally improves retrieval; VTLM similarly improves retrieval but yields relatively incremental gains on VG VQA Japanese.The authors suggest the English–Japanese caption language gap limits VTLM's help on Japanese VQA; they relate MRTM's stronger VQA benefit to fine-grained language–visual alignment.Source: §4.1.3 · pp. 7–8, Table 3 · p. 8
  • Image-pivot pre-training performs better overall than English-pivot pre-training on multilingual retrieval, with a particularly notable advantage for target languages with limited training data.The authors interpret this as stronger transfer of cross-modal mapping across languages.Source: Table 4 · p. 8, §4.1.3 · p. 8
  • Semantically aligned English, German, and Czech words attend to similar image regions, grounding the languages in shared visual context.The visualization shows text-to-image attention from salient words in multilingual captions to the shared image context.Source: Figure 3 · p. 8, §4.1.3 · p. 8

5. Conclusion

UC2 uses machine-translation-augmented multilingual image-to-text pairs to learn cross-lingual cross-modal representations, with images as the primary pivot and English as a complement. The paper reports state-of-the-art performance on two multilingual V+L tasks and strong cross-lingual transfer.

5. Conclusion

  • UC2 is an MT-augmented pre-training model that learns cross-lingual cross-modal representations from large-scale multilingual image-to-text pairs, pivoting primarily on images and complementarily on English.Source: §5 · p. 8
  • Two new pre-training tasks are proposed to improve alignment between vision and different languages.Source: §5 · p. 8
  • The model achieves state-of-the-art performance on two mainstream multilingual V+L tasks and demonstrates strong cross-lingual transfer; future work will expand language families and explore video-text grounding with temporal information.Source: §5 · p. 8

A. Appendix

The supplementary material provides downstream fine-tuning implementation details and additional ablations of the proposed pre-training objectives, which were omitted from the main paper for space reasons.

A. Appendix

  • The supplement provides implementation details for downstream-task fine-tuning.Source: A. Appendix · p. 12
  • It also presents additional ablation studies of the proposed pre-training objectives.Source: A. Appendix · p. 12
  • These materials were omitted from the main paper because of space limits.Source: A. Appendix · p. 12

A.1. Downstream Tasks Details

The appendix specifies fine-tuning procedures for multilingual retrieval and VQA, and identifies the metrics included in the MRTM–MRC ablation table.

A.1. Downstream Tasks Details

  • For multilingual retrieval, UC2 is fine-tuned on English-only data for 30K steps in the zero-shot setting or all-language data for 50K steps, using Multi30K and MSCOCO.Fine-tuning uses batch size 40, two negative pairs per positive pair, Adam with learning rate 1e-4, and an eight-V100 setup.Source: §A.1 · p. 12
  • VQA fine-tuning uses a probability distribution over predefined answers and binary cross-entropy, with separate training schedules for VQA v2.0 and VQA VG Japanese.VQA v2.0 is trained for 7K steps and VQA VG Japanese for 5K steps.Source: §A.1 · p. 12
  • Table 5 reports retrieval Meta-Average over Multi30K and MSCOCO, plus VQA v2.0 test-dev accuracy (VQA EN) and VQA VG Japanese accuracy (VQA JA).Source: Table 5 · p. 12

A.2. Ablation Studies

The appendix examines MRTM against MRC, the effect of pre-training with more languages, and the choice of VTLM pivot language. Results indicate modest benefits from more languages, marginal VQA gains with MRTM, and a lower retrieval meta-average with Chinese rather than English as the VTLM pivot.

A.2. Ablation Studies

  • MRTM yields image-text retrieval performance similar to MRC but marginally improves multilingual VQA, consistent with the proposed role of improving local alignment for region-level recognition and reasoning.Source: §A.2 · p. 12
  • With training time held constant, adding languages from two to three to six slightly improves English and German image-text retrieval.The two-language corpus includes English and German; the three-language corpus adds Chinese, and the six-language corpus uses all languages in the full model.Source: §A.2 · p. 12, Figure 4 · p. 12
  • The authors suggest that multilingual training benefits compensate for reduced training time per language and indicate scalability within a reasonable time frame, while noting that the language-capacity trade-off needs further study.Source: §A.2 · p. 13
  • Changing the VTLM pivot from English to Chinese lowers the retrieval meta-average from 86.2 to 85.5; the authors cite possible noise in machine-translated Chinese captions and call for more experiments.Source: §A.2 · p. 13
Loading 2104.00332v1…