Contrastive Learning for Many-to-many Multilingual Neural Machine Translation

Xiao Pan; Mingxuan Wang; Liwei Wu; Lei Li

Source-linked AI summary

Contrastive Learning for Many-to-many Multilingual Neural Machine Translation

Xiao Pan, Mingxuan Wang, Liwei Wu, Lei Li

arXiv:2105.09501v3May 20, 2021cs.CLcs.LG

TL;DR

Multilingual NMT often lags on non-English directions, motivating a many-to-many system with more unified cross-language representations. mRASP2 combines contrastive learning with augmentation of parallel and monolingual data, achieving broad gains across English-centric, zero-shot, and unsupervised directions, including average improvements of 10+ BLEU.

Problem
Previous multilingual NMT work focuses largely on English-centric directions, while non-English translation quality remains weaker.
Source: §1 · p. 1, §4.2 · p. 6
Method
mRASP2 combines contrastive learning that aligns representations across languages with aligned augmentation of parallel and monolingual data.
Source: §1 · p. 2, §2.2 · p. 3
Results
mRASP2 consistently improves English-centric and non-English directions, outperforming a strong multilingual baseline in 20 WMT directions and achieving 10+ average BLEU improvements across 36 zero-shot and unsupervised directions.
Source: §1 · p. 2, §7 · p. 9
Takeaways & Limitations
The results support the possibility of a true many-to-many multilingual NMT model that works well across translation directions.
Source: §1 · p. 2, §7 · p. 9

Abstract

from arXiv · show

Existing multilingual machine translation approaches mainly focus on English-centric directions, while the non-English directions still lag behind. In this work, we aim to build a many-to-many translation system with an emphasis on the quality of non-English language directions. Our intuition is based on the hypothesis that a universal cross-language representation leads to better multilingual translation performance. To this end, we propose mRASP2, a training method to obtain a single unified multilingual translation model. mRASP2 is empowered by two techniques: a) a contrastive learning scheme to close the gap among representations of different languages, and b) data augmentation on both multiple parallel and monolingual data to further align token representations. For English-centric directions, mRASP2 outperforms existing best unified model and achieves competitive or even better performance than the pre-trained and fine-tuned model mBART on tens of WMT's translation directions. For non-English directions, mRASP2 achieves an improvement of average 10+ BLEU compared with the multilingual Transformer baseline. Code, data and trained models are available at https://github.com/PANXiao1994/mRASP2.

1 Introduction

Multilingual NMT seeks a single model that translates effectively across many language pairs, but prior systems often favor English-centric directions and lose performance as languages are added. mRASP2 addresses this by aligning cross-language representations with contrastive learning and augmentation across parallel and monolingual data.

Multilingual NMT enables one model to serve many languages, with parameter sharing supporting knowledge transfer, low-resource translation, and potentially zero-shot translation.Source: §1 · p. 1
Prior multilingual systems can underperform bilingual baselines, with the gap increasing as model capacity is divided among more languages.Source: §1 · p. 1
mRASP2 targets a unified many-to-many model using English-centric parallel corpora plus monolingual data, aiming to reduce representation gaps between languages.Source: §1 · p. 2
The method combines contrastive learning with aligned augmentation on parallel and monolingual corpora to create pseudo-pairs and further align token representations.Source: Figure 1 · p. 1, §1 · p. 2
mRASP2 improves both English-centric and non-English directions, outperforming a strong multilingual baseline in 20 WMT directions and exceeding mBART on 10 WMT benchmarks.Source: §1 · p. 2
For zero-shot and unsupervised translation, mRASP2 reports strong results across 36 directions, with average improvements exceeding 10 BLEU.Source: §1 · p. 2

2 Methodology

mRASP2 unifies multilingual translation training with contrastive learning and aligned augmentation on parallel and monolingual data. Its objective explicitly aligns representations of similar cross-language sentences while jointly optimizing translation quality.

Multilingual contrastive learning

mRASP2 combines multilingual parallel and monolingual corpora with contrastive learning in one unified training framework.The framework targets a shared multilingual representation and uses both translation and contrastive objectives.Source: §2 · p. 2, §2.2 · p. 2

Multilingual Transformer

The multilingual Transformer uses language-identification tokens on both source and target sides and a 12-layer encoder with a 12-layer decoder.The model dimension is 1024 with 16 heads, alongside Layer Normalization and pre-norm residual connections.Source: §2.1 · p. 2

Multilingual contrastive learning

Contrastive learning pulls representations of translated sentence pairs together and pushes representations of non-matched sentences apart.Positive pairs come from bilingual translation data, while negative sentences are sampled from the other language.Source: §2.2 · pp. 2–3
The joint objective combines token-level cross-entropy translation loss with sentence-level contrastive loss, scaled by average sequence length and weighted by λ.The sequence-length factor compensates for the different granularities of the two losses.Source: §2.2 · p. 3

Aligned augmentation

Aligned Augmentation replaces words with same-meaning synonyms to create pseudo-parallel examples from bilingual data and pseudo self-parallel examples from monolingual data.The augmented examples contribute both translation and contrastive losses; monolingual self-parallel training reconstructs the original sentence.Source: §2.3 · p. 3, Figure 2 · p. 3

3 Experiments

The experiments evaluate mRASP2 across supervised, zero-shot, and unsupervised translation settings using multilingual parallel and monolingual datasets, with BLEU as the primary metric.

PC32 contains 97.6 million sentence pairs across 32 English-centric language pairs.Source: §3.1 · p. 4
Aligned augmentation randomly replaces source-side words with multilingual synonyms at a 90% probability on parallel and monolingual data.Source: §3.1 · p. 4
MC24 contains monolingual text in 24 languages, including three languages absent from PC32, with 1.01 billion sentences overall.The dataset retains PC32 languages and adds Dutch, Polish, and Portuguese.Source: §3.1 · p. 4
mRASP2 is evaluated on supervised directions from WMT and IWSLT, with OPUS-100 used for unavailable language pairs.Source: §3.1 · p. 4
The OPUS-100 zero-shot test set contains 6 languages and 30 translation directions.Source: §3.1 · p. 4
Results are reported primarily with tokenized BLEU, while zero-shot evaluation also reports de-tokenized BLEU using SacreBLEU.Source: §3.1 · pp. 4–5, Table 2 · p. 5, Table 3 · p. 5

4 Experiment Results

mRASP2 delivers consistent gains across supervised, unsupervised, and zero-shot settings, particularly improving non-English directions while preserving English-centric performance.

mRASP2 provides consistent performance gains for supervised and unsupervised English-centric directions as well as non-English directions.Source: §4 · p. 5

Supervised Directions

mRASP2 improves multilingual baselines by a large margin in 10 supervised translation directions.Source: §4.1 · p. 5

Unsupervised Directions

mRASP2 obtains +14.13 BLEU on average for unsupervised directions without explicitly introducing supervision signals.Source: §4.1 · p. 6
mRASP2 achieves reasonable BLEU on Nl↔Pt despite training only on monolingual data from both sides.Source: §4.1 · p. 6

Zero-Shot Translation

mRASP2 significantly outperforms m-Transformer and substantially narrows the gap with pivot-based translation on zero-shot directions.Source: §4.2 · p. 6
mRASP2 improves zero-shot translation by a large margin without losing performance on English-centric directions.Source: §4.2 · p. 6

5 Analysis

The analysis evaluates how contrastive learning and monolingual data affect translation and cross-lingual sentence representations. Across English-centric and non-English retrieval settings, mRASP2 improves alignment, with especially strong gains for Dutch and zero-shot directions.

Ablation Study

Contrastive loss substantially improves zero-shot BLEU while performing comparably with m-Transformer in supervised and unsupervised scenarios.Source: §5.1 · p. 7
Removing contrastive learning causes poor zero-shot performance, indicating its importance for zero-shot directions.Source: §5.1 · p. 7
mRASP2 further improves BLEU in all three scenarios, especially for unsupervised directions, when monolingual data is included.Source: §5.1 · p. 7

Similarity Search

Similarity search measures cross-lingual representation alignment by retrieving each sentence’s nearest neighbor in another language using cosine similarity.Source: §5.2 · p. 7
On 210 non-English directions, mRASP2 increases averaged top-1 similarity-search accuracy, supporting narrower representation gaps across languages.Source: §5.2 · pp. 7–8
mRASP2 improves sentence-retrieval accuracy over m-Transformer for English-centric Tatoeba directions and follows m-Transformer < mRASP2 w/o AA < mRASP2 overall.Source: §5.2 · p. 7, Table 5 · p. 8, Table 6 · p. 8
mRASP2 especially improves Dutch retrieval because it introduces Dutch monolingual data absent from mRASP2 w/o AA.Source: §5.2 · p. 8

Representation Visualization

T-SNE visualizations show mRASP2 draws English, German, and Japanese sentence representations closer than m-Transformer.Source: §5.3 · p. 8

6 Related Work

Related work develops multilingual NMT, zero-shot translation, contrastive learning, and cross-lingual representations. mRASP2 differs by explicitly using contrastive learning to bring multilingual representations together within a many-to-many translation model.

Multilingual Neural Machine Translation

Multilingual NMT seeks a single model for many languages, offering model efficiency, knowledge transfer, and potential zero-shot translation.Source: §6 · p. 8
Earlier multilingual systems often add language-specific components to improve performance on rich-resource languages.Source: §6 · p. 8

Zero-shot Machine Translation

Zero-shot NMT can avoid explicit pivoting, but its performance has lagged behind pivot-based competitors.Source: §6 · p. 8
mRASP2 aims to learn a universal many-to-many model that bridges cross-language representations, unlike prior zero-shot approaches.Source: §6 · p. 9