Source-linked AI summary
A Multilingual View of Unsupervised Machine Translation
Xavier Garcia, Pierre Foret, Thibault Sellam, Ankur P. Parikh
TL;DR
The paper addresses multilingual translation when the target pair lacks direct or multi-step parallel supervision, including a novel setting with auxiliary parallel data for only one pair language. The framework derives multi-way back-translation for monolingual data and a cross-translation loss that uses auxiliary parallel data to enforce cross-language consistency. The approach offers quantifiable gains over existing state-of-the-art unsupervised models on En–Ro, En–Fr, and En–De, with better scores for almost all translation directions.
Problem
The paper addresses multilingual translation when the target pair lacks direct or multi-step parallel supervision, including a novel setting with auxiliary parallel data for only one pair language.
Method
The framework derives multi-way back-translation for monolingual data and a cross-translation loss that uses auxiliary parallel data to enforce cross-language consistency.
Results
The approach offers quantifiable gains over existing state-of-the-art unsupervised models on En–Ro, En–Fr, and En–De, with better scores for almost all translation directions.
Takeaways & Limitations
Multilinguality and auxiliary parallel data provide quantifiable gains over strong baselines within the evaluated translation settings.
Takeaways & Limitations
The framework relies on a conditional-independence assumption, and an alternative agreement-term formulation was unstable without X–Z parallel data.
Abstract
from arXiv · showhide
We present a probabilistic framework for multilingual neural machine translation that encompasses supervised and unsupervised setups, focusing on unsupervised translation. In addition to studying the vanilla case where there is only monolingual data available, we propose a novel setup where one language in the (source, target) pair is not associated with any parallel data, but there may exist auxiliary parallel data that contains the other. This auxiliary data can naturally be utilized in our probabilistic framework via a novel cross-translation loss term. Empirically, we show that our approach results in higher BLEU scores over state-of-the-art unsupervised models on the WMT'14 English-French, WMT'16 English-German, and WMT'16 English-Romanian datasets in most directions. In particular, we obtain a +1.65 BLEU advantage over the best-performing unsupervised model in the Romanian-English direction.
1 Introduction
The paper extends unsupervised neural machine translation to multilingual settings, including a novel setup that uses auxiliary parallel data without directly connecting the target pair. It develops a probabilistic framework with cross-translation objectives and reports gains over strong unsupervised baselines.
- Motivation: Multilingual unsupervised translation has received limited attention, and earlier multilingual systems performed below state-of-the-art bilingual unsupervised models.Supervised multilingual translation and pivot-based zero-shot translation provide related precedents, but zero-shot translation uses parallel supervision through the pivot.
- Contribution: The framework defines strict and looser multilingual unsupervised setups, including cases where auxiliary parallel data contains only one language of the target pair.The target pair remains disconnected, ruling out direct or multi-step supervised translation while permitting auxiliary data to aid rare-language translation.
- Method: The probabilistic formulation yields a multi-way back-translation objective for strict M-UNMT and a cross-translation loss for auxiliary parallel data.The approach also uses auxiliary parallel data for pre-training to build representations better suited to translation.
- Empirical evaluation: The approach improves scores over existing state-of-the-art unsupervised models on English–Romanian, English–French, and English–German in most translation directions.The paper also evaluates ablations of the additional data, loss terms, and auxiliary-language choice.
2 Background and Overview
The background distinguishes supervised translation, which relies on parallel sentence pairs, from unsupervised translation, which learns from separate monolingual corpora. The paper’s overview derives a probabilistic multilingual framework using EM, back-translation, cross-translation, and auxiliary-data pre-training.
- Background: Supervised neural machine translation trains on source–target sentence pairs, whereas unsupervised translation learns from separate monolingual corpora without sentence-aligned equivalents.At inference time, the most likely target sentence is approximated with beam search.
- Background: State-of-the-art unsupervised methods typically pre-train with language-modeling or noisy-reconstruction objectives and fine-tune with back-translation.Back-translation translates a sentence into the other language and back, then penalizes reconstruction error.
- Overview: The paper’s framework models several monolingual corpora as partial observations of multilingual translation triplets, derives a generalized back-translation objective, and adds cross-translation for auxiliary parallel data.The complete procedure uses EM and pre-training to initialize useful representations before likelihood maximization.
3 Multilingual Unsupervised Machine Translation
The paper formulates multilingual unsupervised translation probabilistically for monolingual and auxiliary-parallel settings, deriving objectives with EM. The framework generalizes back-translation, introduces cross-translation consistency, and connects the result to supervised and zero-shot translation.
- 3.1 M-UNMT - Monolingual Data Only: Strict M-UNMT produces an aggregate back-translation objective that generalizes prior bilingual objectives and is optimized with EM.The E-step is approximated by decoding the mode, while the M-step retains the back-translation terms and can be implemented with a gradient update.
- 3 Multilingual Unsupervised Machine Translation: The multilingual model treats monolingual datasets as partial observations of translation triplets and assumes the remaining languages are conditionally independent given any one language.This structural assumption makes the conditional translation models tractable but constrains the model’s representation of multilingual dependencies.
- 3.2 Auxiliary parallel data: The auxiliary-parallel extension introduces cross-translation terms that translate through the auxiliary language and enforce consistency across distinct language pairs.The terms include expectations corresponding to Y-to-Z-to-X and X-to-Z-to-Y translation paths.
- 3.3 Connections with supervised and zero shot methods: The same probabilistic framework recovers supervised translation by conditioning on observed parallel data and relates cross-translation to agreement objectives in zero-shot translation.The zero-shot agreement formulation was unstable in the authors’ setting, which lacked X–Z parallel data available in the compared setup.
4 Training algorithms
The training procedure uses MASS and parallel cross-entropy objectives for pre-training, followed by EM-derived fine-tuning with back-translation or cross-translation losses.
- 4 Training algorithms: The training algorithms are presented as an end-to-end procedure in which pre-training precedes EM-based fine-tuning.The EM procedure is used after initialization to refine the intermediate translation model.
- 4.1 Pre-training: Pre-training initializes the translation model with MASS on monolingual data and cross-entropy translation on auxiliary parallel data.MASS masks contiguous input segments and reconstructs them, while parallel batches use both translation directions.
- 4.2 Fine-tuning: The algorithms initialize parameters from pre-training and iterate over datasets until convergence, decoding through each alternative language for multilingual objectives.Monolingual batches generate reconstructions through every other language, while parallel batches use an intermediate language for cross-translation.
- 4.2 Fine-tuning: The fine-tuning phase selects monolingual or bilingual datasets, applying back-translation for monolingual batches and cross-translation terms in both directions for bilingual batches.Each computed loss produces a parameter update.
5 Experiments
Experiments evaluate M-UNMT across three multilingual triplets using standardized corpora, shared Transformer components, and BLEU-based comparisons with strong unsupervised baselines. Models using auxiliary data achieve better scores in almost all translation directions and outperform competing methods that also use auxiliary parallel data.
- 5 Experiments: Experiments use English-French-Romanian, English-Czech-German, and English-Spanish-French triplets with parallel data linking English to one auxiliary language.The unsupervised directions were selected to enable comparison with recent prior work.
- 5.1 Datasets and preprocessing: The study selects 10 million monolingual lines per language, applies Moses preprocessing, and builds 64,000-token SentencePiece vocabularies.News Crawl supplies monolingual data, while UN Corpus, French-English Gigaword, and CzEng 1.7 supply the listed parallel corpora.
- 5.2 Model architectures: The models use a shared six-layer Transformer encoder and shared decoder parameters, with decoder-side language embeddings and language-specific attention-head output transformations.The Transformer uses hidden size 1024 and a 4096-unit feedforward filter.
- 5.4 Results: Evaluation reports tokenized BLEU for the benchmark directions and additionally provides sacreBLEU scores for French and German on detokenized text.Romanian sacreBLEU scores are omitted because common preprocessing is not natively supported.
- 5.4 Results: The comparison includes XLM, MASS, D2GPo, Artetxe et al. (2019), Ren et al. (2019), mBART, and concurrent auxiliary-data methods.The paper also reports an M-UNMT model using only pre-training, without back-translation or cross-translation.
- 5.4 Results: Auxiliary-data models achieve better scores in almost all translation directions and outperform all listed baselines that also leverage auxiliary parallel data.The results indicate that both the additional data and the chosen objectives contribute to the improved performance.
6 Ablations
Ablations show that auxiliary parallel data improves performance, that gains are strongest when used in both training phases, and that multilingual cross-translation objectives improve BLEU despite higher back-translation loss. Auxiliary-language choice also matters, although any tested auxiliary parallel data remains beneficial relative to baselines.
- Impact of the auxiliary data: The ablations compare four training configurations that vary whether auxiliary parallel data is available during pre-training, fine-tuning, both, or neither.They also include original MASS results and the supervised WMT’16 Romanian-English model for comparison.
- Impact of the auxiliary data: Auxiliary data produces superior performance, including scores surpassing the supervised Sennrich et al. (2016) results, with the strongest performance when used in both training phases.Gains can arise during either pre-training or fine-tuning, but combining both uses performs best.
- Impact of the additional objectives: The Full fine-tuning configuration dominates the other approaches in Ro ´ En, while BT and M-BT plateau about 1 BLEU below Full in En ´ Ro.BT fine-tuning uses only bilingual back-translation, M-BT adds multi-way back-translation, and Full also uses cross-translation.
- Impact of the additional objectives: The BT configuration reaches the lowest back-translation loss but not the highest BLEU, indicating that multilingual objectives improve translation quality while increasing back-translation errors.The authors characterize multilinguality as acting like a regularizer.
- Impact of the choice of auxiliary language: French and Spanish auxiliary languages yield similar BLEU scores, whereas Czech performs worse, showing that auxiliary-language choice affects optimal performance.The Czech configuration still outperforms baselines, so any tested auxiliary parallel data remains valuable.
7 Conclusion and Future Work
The paper concludes that multilinguality and auxiliary parallel data provide quantifiable gains over strong unsupervised translation baselines, and identifies massively multilingual unsupervised translation as future work.
- 7 Conclusion and Future Work: Multilinguality and auxiliary parallel data offer quantifiable gains over strong unsupervised machine translation baselines.The authors describe their approach as a simple multilingual extension of unsupervised machine translation.
- 7 Conclusion and Future Work: The authors propose exploring massively multilingual unsupervised machine translation in future work.