Source-linked AI summary
Contrastive Learning for Many-to-many Multilingual Neural Machine Translation
Xiao Pan, Mingxuan Wang, Liwei Wu, Lei Li
TL;DR
Multilingual NMT often lags on non-English directions, motivating a many-to-many system with more unified cross-language representations. mRASP2 combines contrastive learning with augmentation of parallel and monolingual data, achieving broad gains across English-centric, zero-shot, and unsupervised directions, including average improvements of 10+ BLEU.
Problem
Previous multilingual NMT work focuses largely on English-centric directions, while non-English translation quality remains weaker.
Method
mRASP2 combines contrastive learning that aligns representations across languages with aligned augmentation of parallel and monolingual data.
Results
mRASP2 consistently improves English-centric and non-English directions, outperforming a strong multilingual baseline in 20 WMT directions and achieving 10+ average BLEU improvements across 36 zero-shot and unsupervised directions.
Takeaways & Limitations
The results support the possibility of a true many-to-many multilingual NMT model that works well across translation directions.
Abstract
from arXiv · showhide
Existing multilingual machine translation approaches mainly focus on English-centric directions, while the non-English directions still lag behind. In this work, we aim to build a many-to-many translation system with an emphasis on the quality of non-English language directions. Our intuition is based on the hypothesis that a universal cross-language representation leads to better multilingual translation performance. To this end, we propose mRASP2, a training method to obtain a single unified multilingual translation model. mRASP2 is empowered by two techniques: a) a contrastive learning scheme to close the gap among representations of different languages, and b) data augmentation on both multiple parallel and monolingual data to further align token representations. For English-centric directions, mRASP2 outperforms existing best unified model and achieves competitive or even better performance than the pre-trained and fine-tuned model mBART on tens of WMT's translation directions. For non-English directions, mRASP2 achieves an improvement of average 10+ BLEU compared with the multilingual Transformer baseline. Code, data and trained models are available at https://github.com/PANXiao1994/mRASP2.
1 Introduction
Multilingual NMT seeks a single model that translates effectively across many language pairs, but prior systems often favor English-centric directions and lose performance as languages are added. mRASP2 addresses this by aligning cross-language representations with contrastive learning and augmentation across parallel and monolingual data.
- Multilingual NMT enables one model to serve many languages, with parameter sharing supporting knowledge transfer, low-resource translation, and potentially zero-shot translation.
- Prior multilingual systems can underperform bilingual baselines, with the gap increasing as model capacity is divided among more languages.
- mRASP2 targets a unified many-to-many model using English-centric parallel corpora plus monolingual data, aiming to reduce representation gaps between languages.
- The method combines contrastive learning with aligned augmentation on parallel and monolingual corpora to create pseudo-pairs and further align token representations.
- mRASP2 improves both English-centric and non-English directions, outperforming a strong multilingual baseline in 20 WMT directions and exceeding mBART on 10 WMT benchmarks.
- For zero-shot and unsupervised translation, mRASP2 reports strong results across 36 directions, with average improvements exceeding 10 BLEU.
2 Methodology
mRASP2 unifies multilingual translation training with contrastive learning and aligned augmentation on parallel and monolingual data. Its objective explicitly aligns representations of similar cross-language sentences while jointly optimizing translation quality.
- Multilingual contrastive learning: mRASP2 combines multilingual parallel and monolingual corpora with contrastive learning in one unified training framework.The framework targets a shared multilingual representation and uses both translation and contrastive objectives.
- Multilingual Transformer: The multilingual Transformer uses language-identification tokens on both source and target sides and a 12-layer encoder with a 12-layer decoder.The model dimension is 1024 with 16 heads, alongside Layer Normalization and pre-norm residual connections.
- Multilingual contrastive learning: Contrastive learning pulls representations of translated sentence pairs together and pushes representations of non-matched sentences apart.Positive pairs come from bilingual translation data, while negative sentences are sampled from the other language.
- Multilingual contrastive learning: The joint objective combines token-level cross-entropy translation loss with sentence-level contrastive loss, scaled by average sequence length and weighted by λ.The sequence-length factor compensates for the different granularities of the two losses.
- Aligned augmentation: Aligned Augmentation replaces words with same-meaning synonyms to create pseudo-parallel examples from bilingual data and pseudo self-parallel examples from monolingual data.The augmented examples contribute both translation and contrastive losses; monolingual self-parallel training reconstructs the original sentence.
3 Experiments
The experiments evaluate mRASP2 across supervised, zero-shot, and unsupervised translation settings using multilingual parallel and monolingual datasets, with BLEU as the primary metric.
- PC32 contains 97.6 million sentence pairs across 32 English-centric language pairs.
- Aligned augmentation randomly replaces source-side words with multilingual synonyms at a 90% probability on parallel and monolingual data.
- MC24 contains monolingual text in 24 languages, including three languages absent from PC32, with 1.01 billion sentences overall.The dataset retains PC32 languages and adds Dutch, Polish, and Portuguese.
- mRASP2 is evaluated on supervised directions from WMT and IWSLT, with OPUS-100 used for unavailable language pairs.
- The OPUS-100 zero-shot test set contains 6 languages and 30 translation directions.
- Results are reported primarily with tokenized BLEU, while zero-shot evaluation also reports de-tokenized BLEU using SacreBLEU.
4 Experiment Results
mRASP2 delivers consistent gains across supervised, unsupervised, and zero-shot settings, particularly improving non-English directions while preserving English-centric performance.
- mRASP2 provides consistent performance gains for supervised and unsupervised English-centric directions as well as non-English directions.
- Supervised Directions: mRASP2 improves multilingual baselines by a large margin in 10 supervised translation directions.
- Unsupervised Directions: mRASP2 obtains +14.13 BLEU on average for unsupervised directions without explicitly introducing supervision signals.
- Unsupervised Directions: mRASP2 achieves reasonable BLEU on Nl↔Pt despite training only on monolingual data from both sides.
- Zero-Shot Translation: mRASP2 significantly outperforms m-Transformer and substantially narrows the gap with pivot-based translation on zero-shot directions.
- Zero-Shot Translation: mRASP2 improves zero-shot translation by a large margin without losing performance on English-centric directions.
5 Analysis
The analysis evaluates how contrastive learning and monolingual data affect translation and cross-lingual sentence representations. Across English-centric and non-English retrieval settings, mRASP2 improves alignment, with especially strong gains for Dutch and zero-shot directions.
- Ablation Study: Contrastive loss substantially improves zero-shot BLEU while performing comparably with m-Transformer in supervised and unsupervised scenarios.
- Ablation Study: Removing contrastive learning causes poor zero-shot performance, indicating its importance for zero-shot directions.
- Ablation Study: mRASP2 further improves BLEU in all three scenarios, especially for unsupervised directions, when monolingual data is included.
- Similarity Search: Similarity search measures cross-lingual representation alignment by retrieving each sentence’s nearest neighbor in another language using cosine similarity.
- Similarity Search: On 210 non-English directions, mRASP2 increases averaged top-1 similarity-search accuracy, supporting narrower representation gaps across languages.
- Similarity Search: mRASP2 improves sentence-retrieval accuracy over m-Transformer for English-centric Tatoeba directions and follows m-Transformer < mRASP2 w/o AA < mRASP2 overall.
- Similarity Search: mRASP2 especially improves Dutch retrieval because it introduces Dutch monolingual data absent from mRASP2 w/o AA.
- Representation Visualization: T-SNE visualizations show mRASP2 draws English, German, and Japanese sentence representations closer than m-Transformer.
6 Related Work
Related work develops multilingual NMT, zero-shot translation, contrastive learning, and cross-lingual representations. mRASP2 differs by explicitly using contrastive learning to bring multilingual representations together within a many-to-many translation model.
- Multilingual Neural Machine Translation: Multilingual NMT seeks a single model for many languages, offering model efficiency, knowledge transfer, and potential zero-shot translation.
- Multilingual Neural Machine Translation: Earlier multilingual systems often add language-specific components to improve performance on rich-resource languages.
- Zero-shot Machine Translation: Zero-shot NMT can avoid explicit pivoting, but its performance has lagged behind pivot-based competitors.
- Zero-shot Machine Translation: mRASP2 aims to learn a universal many-to-many model that bridges cross-language representations, unlike prior zero-shot approaches.
- Contrastive Learning: Contrastive learning has been applied to sentence representation learning through sentence-level augmentation strategies.
- Cross-lingual Representation: Multilingual masked language models primarily study unsupervised cross-lingual representation learning, while TLM uses concatenated parallel sentences as input.
- Cross-lingual Representation: mRASP2 instead exploits supervised parallel data by pulling representations of similar multilingual sentences closer.
7 Conclusion
The paper concludes that contrastive learning improves zero-shot translation and, with monolingual data, yields strong performance across multilingual translation directions. Its analyses associate these gains with reduced cross-language representation gaps and motivate scaling to more languages.
- Combining contrastive learning with additional monolingual data produces substantial improvements across multilingual translation directions.
- Analyses and visualizations find that mRASP2 tends to close representation gaps between languages.
- The authors identify the possibility of a true many-to-many multilingual NMT model that works well across translation directions.
B Details of Evaluation Results
The paper reports detailed evaluation results across a wide range of test sets.
- Detailed evaluation results are provided for a wide range of test sets.
B.1 Results on OPUS-100
The section points readers to detailed results for the OPUS-100 zero-shot evaluation set.
- Detailed results are provided for the OPUS-100 zero-shot evaluation set.
- The OPUS-100 results are organized as a detailed evaluation-set report.
- The referenced evaluation concerns zero-shot performance on OPUS-100.
B.2 Results on WMT
The section reports WMT evaluation results and illustrates two examples of sentences after applying AA.
- Detailed results are provided for the WMT evaluation set.
- Two results of sentences after AA are shown in Figure 6.
- Figure 6 presents examples of sentences after AA.
D Details of MC24
This section gives implementation and visualization details for MC24 and AA, alongside referenced OPUS-100 and WMT result tables.
- MC24 is described in detail in Table 10.
- Table 7 presents parallel sentences distributed across English, German, and Japanese.
- Tables 8 and 9 report OPUS-100 zero-shot and public WMT BLEU results, respectively, with specified mRASP and ablation settings.
- Figure 5 shows multi-way parallel sentence representations in the mRASP2 representation space.
- Figure 6 contains two examples of sentences together with their noised versions after AA.
- Table 10 uses '?' for missing data and '/' when a language is absent from the synonym dictionary.