Source-linked AI summary
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
Biao Zhang, Philip Williams, Ivan Titov, Rico Sennrich
TL;DR
Massively multilingual NMT often underperforms bilingual models because of limited capacity and produces poor zero-shot translations through off-target errors. The paper addresses these problems with language-aware, deeper models and random online backtranslation, narrowing the bilingual gap and improving zero-shot performance by ~10 BLEU.
Problem
Massively multilingual NMT suffers from insufficient modeling capacity and off-target translations on zero-shot directions.
Method
The paper combines deeper, language-aware NMT components with random online backtranslation for unseen training language pairs.
Results
The approach narrows the performance gap with bilingual models, reduces off-target translations by ∼50%, and improves zero-shot performance by ∼10 BLEU.
Takeaways & Limitations
The experiments demonstrate the feasibility of massively zero-shot translation with multilingual NMT on OPUS-100.
Takeaways & Limitations
The one-to-many training and test sets were not aggressively filtered for sentence overlap, so their results are not directly comparable to many-to-many results.
Abstract
from arXiv · showhide
Massively multilingual models for neural machine translation (NMT) are theoretically attractive, but often underperform bilingual models and deliver poor zero-shot translations. In this paper, we explore ways to improve them. We argue that multilingual NMT requires stronger modeling capacity to support language pairs with varying typological characteristics, and overcome this bottleneck via language-specific components and deepening NMT architectures. We identify the off-target translation issue (i.e. translating into a wrong target language) as the major source of the inferior zero-shot performance, and propose random online backtranslation to enforce the translation of unseen training language pairs. Experiments on OPUS-100 (a novel multilingual dataset with 100 languages) show that our approach substantially narrows the performance gap with bilingual models in both one-to-many and many-to-many settings, and improves zero-shot performance by ~10 BLEU, approaching conventional pivot-based methods.
1 Introduction
Massively multilingual NMT offers deployment, transfer, low-resource, and zero-shot benefits but underperforms bilingual models because limited capacity must serve diverse directions. The paper addresses this with language-specific components, deeper architectures, and random online backtranslation (ROBT), which targets off-target zero-shot translation.
- 1 Introduction: The baseline often produces off-target French→German zero-shot translations by copying the source sentence or translating into English instead of German.This illustrates the broader problem that multilingual models may ignore target-language information when no parallel training data exists for the language pair.
- 1 Introduction: The proposed architecture uses language-aware layer normalization and an encoder–decoder linear transformation to relax representation constraints and induce language-specific translation correspondences.The paper also investigates deep NMT architectures to further reduce the performance gap with bilingual methods.
- 1 Introduction: Experiments use OPUS-100, a 55M-pair English-centric dataset covering 100 languages, under one-to-many and many-to-many translation settings.The dataset was collected from OPUS and released to facilitate future research.
- 1 Introduction: Increasing multilingual model capacity yields large improvements, narrows the bilingual-model gap, and benefits low-resource translation most.Language-specific modeling and deep NMT architectures slightly improve zero-shot translation but do not alleviate off-target translation.
- 1 Introduction: ROBT reduces off-target translations by ∼50% and improves zero-shot performance by ∼10 BLEU, approaching the conventional pivot-based method.ROBT finetuning converges within a few thousand steps.
2 Related Work
Prior multilingual NMT work evolved from language-specific multitask architectures toward shared components and massively multilingual models, while zero-shot research addressed wrong-target-language errors through cross-lingual methods and artificial parallel data. ROBT belongs to the latter line, using online backtranslation with randomly selected intermediate languages.
- Multilingual NMT: Multilingual NMT began with multitask approaches that shared encoders or attention mechanisms, but language-specific encoders or decoders limited scalability.Dong et al. (2015) used a shared encoder for one-to-many translation, while Firat et al. (2016a) shared attention for many-to-many translation; Lee et al. (2017) later used character-level inputs with a shared encoder for many-to-one translation.
- Zero-shot translation: Zero-shot multilingual NMT often translates into the wrong target language, motivating cross-lingual regularization and artificial parallel-data approaches.The paper links this failure to the ‘missing ingredient problem’ (Arivazhagan et al., 2019a) and spurious correlation (Gu et al., 2019), while identifying cross-lingual methods and backtranslation or pivot-based translation as two solution categories.
- Zero-shot translation: ROBT extends backtranslation-based zero-shot methods by performing online backtranslation at each training step with randomly selected intermediate languages.Unlike Gu et al. (2019) and Lakew et al. (2019), ROBT avoids decoding the whole training set for each zero-shot direction.
- Massively multilingual translation: The work continues research on massively multilingual translation by developing approaches intended to improve both multilingual and zero-shot performance.Aharoni et al. (2019) demonstrated the feasibility of massively multilingual NMT and reported encouraging results; Arivazhagan et al. (2019b) independently also examined increasing model capacity with deep architectures.
3 Multilingual NMT
The multilingual NMT baseline prepends a target-language token to each source sentence, enabling one model to translate into multiple languages. The system uses a six-layer Transformer encoder-decoder backbone with residual connections and layer normalization.
- Multilingual NMT approach: A single multilingual NMT model is trained by prepending a token specifying the target language to each source sentence, such as <2EN> for English [Johnson et al., 2017].This modified multilingual dataset supports translation into multiple target languages.
- Transformer backbone: The baseline uses the Transformer as its backbone because of its superior multilingual performance [Vaswani et al., 2017; Lakew et al., 2018].The encoder and decoder each contain six layers; decoder layers additionally include cross-attention over the source representation.
- Transformer backbone: Each Transformer sublayer uses a residual connection followed by layer normalization, which accelerates model convergence [He et al., 2015; Ba et al., 2016].The encoder layers combine self-attention with point-wise feedforward sublayers, while the decoder adds cross-attention.
4 Approach
The approach addresses multilingual NMT’s insufficient modeling capacity and zero-shot off-target translation through deeper architectures, language-aware components, and scalable random online backtranslation.
- Motivation: The approach targets insufficient modeling capacity and off-target translation, which become severe as multilingual NMT scales to many languages.The authors identify these as the two central drawbacks of massively multilingual NMT.
- Deep Transformer: Deep Transformer increases model depth using depth-scaled initialization to improve multilingual translation capacity.The method is motivated by deeper models’ ability to learn more abstract representations and complex dependencies.
- Language-aware Layer Normalization: Language-aware Layer Normalization conditions normalization on the target-language token, relaxing the shared Gaussian-space constraint across languages.The formula is applied to all normalization layers, while conditioning on source-language information is left for future work.
- Language-aware Linear Transformation: Language-aware Linear Transformation inserts a target-language-specific transformation between encoder and decoder to express more flexible translation relationships.Adding a target language requires only one weight matrix, balancing expressivity and scalability.
- Random Online Backtranslation: 9702 zero-shot directions make decoding the full training set for each pair impractical, so ROBT uses online backtranslation with randomly sampled intermediate languages.Batch-based greedy decoding reduces computational cost, and the method still benefits from translation signals despite initially imperfect online translations.
5 OPUS-100
OPUS-100 is an English-centric multilingual NMT dataset with 100 languages and up to 1M training pairs per language pair, designed to support many-to-many and zero-shot evaluation. It contains approximately 55M sentence pairs sampled from OPUS without domain balancing, with held-out data for validation, testing, and 15 non-English zero-shot pairings.
- Dataset construction: All training pairs include English, so language pairs without English require zero-shot translation or pivoting through English.This English-centric design follows Aharoni et al. (2019).
- Dataset construction: OPUS-100 covers 100 languages, including English, with up to 1M training pairs for each language pair.Languages were selected according to the volume of available parallel data in OPUS.
- Data composition: The dataset combines OPUS corpora ranging from movie subtitles to Bible documentation without curating or balancing domains.Data were downloaded for each language pair, concatenated, and filtered to avoid monolingual sentence overlap across training, validation, and test sets.
- Dataset statistics: Approximately 55M sentence pairs comprise OPUS-100; 44 of 99 language pairs have 1M training pairs, 73 have at least 100k, and 95 have at least 10k.The dataset also includes 2000 validation and 2000 test pairs per language pair.
- Evaluation setup: Zero-shot evaluation uses 2000 test sentence pairs for each of 15 pairings among Arabic, Chinese, Dutch, French, German, and Russian, excluding sentences already in OPUS-100.These pairings do not include English and therefore test translation between non-English languages.
6 Experiments
Experiments on OPUS-100 show that language-aware components and deeper Transformers substantially improve multilingual NMT, narrowing its gap with bilingual models, especially for low-resource and many-to-many translation. Random online backtranslation largely resolves off-target zero-shot translation, raising quality toward pivot-based methods while incurring modest supervised-translation costs.
- One-to-many ablations: Language-aware components and depth address capacity limitations: combining LALN and LALT adds 3.37 BLEU94 and 55.3% WR, while deepening adds 1.88 BLEU94.Deep Transformers are less strong than LALN+LALT in BLEU at similar parameter counts but improve more consistently across language pairs by 6.4% WR.
- One-to-many results: 29.60 BLEU94 and 21.23 BLEU4 are achieved by the full approach, beating the multilingual baseline on 92.6% of tasks and surpassing the base bilingual model by 0.33 BLEU4.The approach narrows the multilingual–bilingual gap from 20.90 to 21.23 BLEU4, although a similarly deep bilingual model remains 1.52 BLEU4 better.
- Many-to-many results: In many-to-many translation, capacity issues are more severe for English→X, but deep Transformer plus LALN and LALT reduces the bilingual gap from -4.93 to -0.45 BLEU4.For X→English, multitask learning already exceeds bilingual NMT by 2.13 BLEU4, and enhanced capacity increases the margin to 4.80 BLEU4.
- Data scale and direction: Language-aware modeling helps low-resource English→X pairs most, adding 5.82 BLEU versus 1.37/3.11 BLEU for high/medium-resource pairs, while depth improves both directions similarly.Integrating both approaches gives the best performance and positive transfer to low-resource pairs; English→X remains weaker than X→English because training data targets English disproportionately.
- Zero-shot translation: ROBT converges within a few thousand online-backtranslation steps, but costs approximately 0.50 BLEU94 and 4% WR on supervised English↔X translation.Greater model capacity also improves zero-shot language accuracy and BLEUzero before and after ROBT, indicating increased robustness.
7 Conclusion and Future Work
The paper improves massively multilingual NMT by increasing model capacity and correcting off-target zero-shot translations with random online backtranslation. Experiments on the released OPUS-100 dataset show narrowed gaps with bilingual and pivot-based methods, while future work targets lighter language-aware models and stronger zero-shot improvements.
- Conclusion: Multilingual NMT suffers from weak capacity, addressed by deepening the Transformer and devising language-aware neural models.The paper also identifies off-target translations on zero-shot directions and proposes random online backtranslation to correct them.
- Conclusion: The proposed approaches substantially increase translation performance, narrowing the gap with bilingual NMT models and pivot-based methods on OPUS-100.OPUS-100 contains 100 languages and around 55M sentence pairs.
- Future Work: Future work will develop lightweight alternatives to LALT and explore generative modeling strategies to surpass ROBT’s upper bound for larger zero-shot improvements.The cited generative modeling strategies include Zhang et al. (2016), Su et al. (2018), García et al. (2020), and Zheng et al. (2020).
A OPUS-100: The OPUS Multilingual Dataset
This section introduces OPUS-100 as an English-centric multilingual dataset and summarizes its language coverage and sentence-pair counts.
- A OPUS-100: The OPUS Multilingual Dataset: Table 8 lists the non-English languages included in OPUS-100 and the number of sentence pairs for each.The dataset is English-centric.
B Model Settings
The models use Adam-style optimization with label smoothing, scheduled learning rates, dropout, and token-based batching. Multilingual training uses a lower initial learning rate and deeper models receive higher dropout rates.
- Optimization: Multilingual models use an initial learning rate of 0.5 versus 1.0 for bilingual models to stabilize training.Training also uses label smoothing of 0.1 and a 4k-step learning-rate warmup.
- Regularization: Dropout is applied to residual layers and attention weights at 0.1/0.1 for 6-layer Transformers and 0.3/0.2 for deeper models.
- Batching: Training batches contain roughly 50k target tokens grouped from sentence pairs, except for bilingual models.
- Dataset: The English-centric multilingual dataset includes 1,000,000 training, 2,000 validation, and 2,000 test sentence pairs for Lithuanian.