Source-linked AI summary
Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information
Zehui Lin, Xiao Pan, Mingxuan Wang, Xipeng Qiu, Jiangtao Feng, Hao Zhou, Lei Li
TL;DR
The paper asks whether one universal MT model can seed specialized models for arbitrary language pairs, addressing gaps in prior MT pre-training objectives and coverage. It proposes mRASP, which uses random aligned substitution in multilingual pre-training before downstream fine-tuning. Experiments report gains across resource levels, including exotic pairs whose languages were absent from pre-training.
Problem
Prior MT pre-training used objectives different from translation and had not shown one model improving arbitrary language pairs, including rich-resource English–French.
Source: §1 · p. 1Method
mRASP pre-trains a multilingual MT model with random aligned substitution, then fine-tunes it on downstream language pairs using a consistent architecture and objective.
Source: §1 · pp. 1–2Results
Across low-, medium-, rich-resource and exotic settings, mRASP improves over direct training, with gains up to +22 BLEU points for extremely low-resource pairs (<100k).
Source: §1 · p. 2Takeaways & Limitations
Even when neither language appears in pre-training, fine-tuning can produce better exotic-full translation than direct training, with reported gains of +3.3 to +14.1 BLEU.
Source: §1 · p. 2
Abstract
from arXiv · showhide
We investigate the following question for machine translation (MT): can we develop a single universal MT model to serve as the common seed and obtain derivative and improved models on arbitrary language pairs? We propose mRASP, an approach to pre-train a universal multilingual neural machine translation model. Our key idea in mRASP is its novel technique of random aligned substitution, which brings words and phrases with similar meanings across multiple languages closer in the representation space. We pre-train a mRASP model on 32 language pairs jointly with only public datasets. The model is then fine-tuned on downstream language pairs to obtain specialized MT models. We carry out extensive experiments on 42 translation directions across a diverse settings, including low, medium, rich resource, and as well as transferring to exotic language pairs. Experimental results demonstrate that mRASP achieves significant performance improvement compared to directly training on those target pairs. It is the first time to verify that multiple low-resource language pairs can be utilized to improve rich resource MT. Surprisingly, mRASP is even able to improve the translation quality on exotic languages that never occur in the pre-training corpus. Code, data, and pre-trained models are available at https://github.com/linzehui/mRASP.
1 Introduction
The introduction asks whether one multilingual MT model can serve as a shared starting point for specialized models on arbitrary language pairs. It presents mRASP’s aligned-substitution approach and reports gains across resource levels and exotic pairs.
1 Introduction
- The paper identifies a gap: prior MT pre-training used objectives that differ from translation and did not offer one model shown to improve arbitrary pairs, including rich-resource English–French.It also notes that BERT-like models can require sophisticated techniques for fine-tuning.Source: §1 · p. 1
- mRASP pre-trains one multilingual MT model as a common initialization for fine-tuning on arbitrary language pairs, improving over direct training on downstream parallel data.The approach uses the same model architecture and training objective for pre-training and fine-tuning.Source: §1 · p. 1
- Random aligned substitution trains code-switched sentence pairs with translation loss to bring representations across languages closer.The method replaces words using aligned vocabulary across languages.Source: §1 · p. 2
- Up to +22 BLEU points on extremely low-resource pairs (<100k), with gains persisting as data scales to rich-resource tasks and exotic-full pairs (+3.3 to +14.1 BLEU).The comparisons are against models directly trained on the downstream pairs.Source: §1 · p. 2
- The authors describe mRASP as efficient in its use of multilingual parallel data, using several hundred million bilingual pairs rather than hundreds of billions of monolingual or cross-lingual sentences.They suggest that matching pre-training and fine-tuning objectives contributes to performance.Source: §1 · p. 2
- Experiments cover 42 translation directions; reported examples include 14.1 BLEU for Dutch–Portuguese with 12k sentence pairs and 44.3 BLEU on WMT14 English–French.The pre-trained model uses parallel corpora in 32 languages, without the much larger monolingual corpora used by other methods.Source: §1 · p. 2
2 Methodology
mRASP jointly pre-trains a multilingual translation model and fine-tunes it for downstream language pairs. Random aligned substitution trains on translated word substitutions to bring words with similar meanings closer in representation space.
2.1 mRASP
- A many-to-many model jointly learns translation across parallel datasets, using a 6-layer-per-side Transformer-large architecture with 1,024 dimensions and 16 heads.Artificial source and target language tokens identify the translation direction.Source: §2.1 · pp. 2–3
- Random aligned substitution targets shared-vocabulary gaps by training on parallel sentences with cross-language word substitutions, bringing same-meaning words closer in representation space.The model trains substituted sentences with the same translation loss as the originals.Source: §2.1 · p. 3, Figure 1 · p. 3
2.2 Pre-training Data
- Pre-training uses PC32 (Parallel Corpus 32), with English anchoring 64 directed translation pairs comprising 197M sentence pairs; RAS uses ground-truth En-X bilingual dictionaries only where available.Source: §2.2 · pp. 3–4
2.3 Pre-training Details
- Training uses 32k shared BPE merges and a 64,808-token vocabulary; RAS replaces source words with 30% probability from top-1,000 dictionary entries, and pre-training runs 150,000 steps.The vocabulary is balanced by oversampling lower-resource language corpora.Source: §2.3 · p. 4
3 Experiments
The experiments compare mRASP with direct training and other pre-training approaches across resource levels, then test transfer to language pairs absent from pre-training. mRASP improves translation across low-, medium-, and rich-resource settings, including exotic directions.
3 Experiments
- The section reports consistent gains across scenarios and compares mRASP with existing pre-training methods on En→Ro.It also reports gains when combining mRASP with back-translation.Source: §3 · p. 4, §3.2 · p. 5
3.1 Experiment Settings
- The experiments use 14 parallel-corpus pairs, categorized by dataset size, and compare pre-training with direct training on downstream bilingual data.The stated categories are extremely low (<100K), low (>100K and <1M), medium (>1M and <10M), and rich (>10M) resource.Source: §3.1 · p. 5
3.2 Main Results
- mRASP improves low- and medium-resource results, gains 1.0 BLEU on En→De, and gains 1.1 BLEU on rich-resource En→Fr.The reported gain on En→De is for the newstest14 benchmark.Source: §3.2 · p. 5, Table 2 · p. 5
- Combining mRASP with back-translation boosts Ro→En performance by up to 2 BLEU points, suggesting the approaches are complementary.The compared pre-training competitors use substantially more pre-training data.Source: §3.2 · p. 6, Table 4 · p. 6
- Against mBART, mRASP outperforms it on most language pairs; on En-De and En-Fr, it gains 4.3 and 2.9 BLEU over the baseline.The comparison uses the same test sets as mBART.Source: §3.2 · p. 6
3.3 Generalization to Exotic Translation
- mRASP improves performance across exotic translation categories, including Exotic Full Dutch–Portuguese with only 11K pairs, where the direct-training baseline fails to train appropriately.For Exotic Full, neither language occurs in pre-training; the abstract reports gains of +3.3 to +14.1 BLEU for exotic full pairs.Source: §3.3 · p. 6, Table 3 · p. 6
4 Analysis
The analysis attributes most performance gains to pre-training, with fine-tuning adding further gains, and finds that RAS improves translation and brings cross-language representations closer. It also examines the effect of fine-tuning data volume.
The effects with fine-tuning
- Performance mainly comes from pre-training, while fine-tuning adds gains; mRASP without fine-tuning also outperforms the randomly initialized baseline, especially on low-resource data.The authors hypothesize that fine-tuning mainly adjusts the specific language embedding while leaving most other parameters largely unchanged.Source: §4 · p. 7, Table 6 · p. 8
The effectiveness of RAS technique
- RAS consistently improves translation across datasets, with gains up to 2.5+ BLEU over pre-training without alignment information.The comparison is between mRASP with RAS and NA-mRASP without RAS.Source: §4 · p. 7, Table 7 · p. 8
- Average cosine similarity for same-meaning words rises after RAS across both similar and dissimilar language pairs, including En-Zh.The analysis uses the top 1,000 frequent words from the MUSE dictionary and sums their BPE subwords.Source: §4 · pp. 7–8, Figure 3
- After RAS, embedding distributions become closer for both En-Zh and En-Af, with a smaller angle for En-Zh and greater overlap for En-Af.En-Zh represents dissimilar pairs and En-Af similar pairs in the visualization.Source: Figure 2 · p. 7, §4 · p. 8
Fine-tuning Volume
- With 1K En-De parallel pairs, the pre-trained model reaches 24.46 BLEU, while random initialization fails; with 1M pairs, mRASP matches the baseline trained on 4.5M.The baseline does not improve until the fine-tuning dataset reaches 50K pairs.Source: Figure 4 · p. 8, §4 · p. 9
5 Related Works
Prior work spans multilingual NMT, cross-lingual representations, and language-model pre-training; mRASP is distinguished by its sequence-to-sequence NMT focus and alignment regularization.
5 Related Works
- mRASP seeks specialized single-pair models from multilingual pre-training and reports gains for rich-resource language pairs.This contrasts with multilingual NMT models that may have inferior accuracy to individually trained models when many language pairs are involved.Source: §5 · p. 9
- Prior unsupervised pre-training improved natural-language-understanding methods, while mRASP uses more than 110 million sentence pairs to boost downstream NMT.The related-work discussion notes that scaling data is important for successful pre-training.Source: §5 · p. 9
- Cross-lingual representation studies include XLM, which uses multilingual masked language modeling and reports results on XNLI and unsupervised NMT.The passage also describes work extending XLM with multi-task learning to create a universal language encoder.Source: §5 · p. 9
- mRASP differs from cross-lingual representation work as a multilingual sequence-to-sequence model for NMT that adds alignment regularization.The regularization is intended to bridge sentence representations across languages.Source: §5 · p. 9
6 Conclusion
mRASP incorporates word alignment into multilingual NMT pre-training and is evaluated across resource levels and exotic language pairs. The paper reports that alignment bridges language gaps and boosts performance.
6 Conclusion
- Experiments across low-, medium-, rich-resource, and exotic settings demonstrate mRASP's efficacy.The paper also reports analytical experiments on the role of alignment information.Source: §6 · p. 9
- Analytical experiments show that alignment information bridges language gaps and boosts performance.The paper leaves exploration of different alignment approaches for future work.Source: §6 · p. 9
- Future work will explore different alignment approaches and pre-train on larger corpora.The authors state that larger corpora are intended to further boost performance.Source: §6 · p. 9
A.1 Visualization of Word Embedding
The appendix compares word-embedding visualizations with and without RAS across four English-language pairs; the overall distributions become closer after RAS.
A.1 Visualization of Word Embedding
- Across En-Ro, En-Ar, En-Tr, and En-De, overall word-embedding distributions become closer after RAS.Figures 5–8 show the respective English-Romanian, English-Arabic, English-Turkish, and English-German comparisons.Source: §A.1 · p. 12
A.4 Data Description
The pre-training corpus PC32 contains 32 English-centric language pairs totaling 110 million sentence pairs, drawn from multiple public corpus sources.
A.4 Data Description
- PC32 contains 32 English-centric language pairs totaling 110M sentence pairs.Table 10 describes the number of parallel sentence pairs between English and each other language.Source: §A.4 · p. 12, Table 10 · p. 17
- The parallel corpora come from TED, WMT, Europarl, ParaCrawl, OpenSubtitles, and QED.Source: §A.4 · p. 12
- The appendix includes a case study and fine-tuning results on public extremely low-, low-, and medium-resource test sets.These are identified as Tables 8 and 9, respectively.Source: Table 8 · p. 15, Table 9 · p. 16