Source-linked AI summary
Back-translation for Large-Scale Multilingual Machine Translation
Baohao Liao, Shahram Khadivi, Sanjika Hewavitharana
TL;DR
The paper investigates how to build a single multilingual translation system and extend back-translation comparisons from bilingual to multilingual settings. It finds constrained sampling performs best among the tested methods and reports second place in both small tasks.
Problem
The paper investigates building a single multilingual translation system, against the hypothesis that a universal cross-language representation can improve multilingual translation performance.
Source: §1 · p. 1Method
The authors compare beam search and sampling methods for multilingual back-translation, and examine vocabulary size and synthetic-data quantity.
Source: §1 · p. 1, §4.2 · p. 4Results
Takeaways & Limitations
The authors report that smaller vocabularies are a better choice and extensive monolingual English data offers a modest improvement.
Source: §5 · pp. 5–6Time and resource limits constrained what the authors could complete.
Source: §4.3 · p. 5
Abstract
from arXiv · showhide
This paper illustrates our approach to the shared task on large-scale multilingual machine translation in the sixth conference on machine translation (WMT-21). This work aims to build a single multilingual translation system with a hypothesis that a universal cross-language representation leads to better multilingual translation performance. We extend the exploration of different back-translation methods from bilingual translation to multilingual translation. Better performance is obtained by the constrained sampling method, which is different from the finding of the bilingual translation. Besides, we also explore the effect of vocabularies and the amount of synthetic data. Surprisingly, the smaller size of vocabularies perform better, and the extensive monolingual English data offers a modest improvement. We submitted to both the small tasks and achieved the second place.
1 Introduction
The paper explores back-translation methods for multilingual machine translation, alongside vocabulary size and synthetic-data quantity. Its submissions placed second in both WMT-21 small tasks.
- The experiments focus mainly on Small Task 2, which has little training data, and compare back-translation methods, vocabulary sizes, and synthetic-data amounts.Source: §1 · p. 1
- The work examines whether a single multilingual system can support translation across many language pairs.Source: §1 · p. 1
- The submissions placed second in both small tasks, with average hidden-test spBLEU scores of 34.96 for Small Task 1 and 33.34 for Small Task 2.Source: §1 · p. 1
2 Related Work
The paper builds on work developing single multilingual NMT models and extends comparisons of back-translation methods from bilingual to multilingual translation.
- Multilingual NMT research includes language-specific components and single models trained on parallel and monolingual data; this paper pursues the single-model direction for simpler industrial deployment.Source: §2 · p. 1
- Back-translation leverages monolingual data for low-resource pairs, and prior bilingual work compared beam search with constrained and unconstrained sampling.Source: §2 · p. 1
- The paper extends those back-translation comparisons to multilingual settings, where similarities among languages may affect results.Source: §2 · p. 2
3 Experimental Setup
The experiments use organizer-provided parallel and monolingual data for two small tasks, with filtering and temperature sampling for synthetic data. Models use multilingual Transformer setups, and evaluation uses spBLEU.
3.1 Data
- The data cover two small tasks and multiple evaluation sets; synthetic pairs are filtered by sentence length and length ratio, then sampled across languages with temperature T = 5.Source: §3.1 · p. 2
3.2 Model
- The multilingual Transformer models use target-language labels, with Trans_small, Trans_base, and Trans_big architectures; submissions use Trans_base, while Trans_big generates synthetic data.Source: §3.2 · p. 2
3.3 Optimization and Evaluation
- Training schedules differ for ablations and final submissions; models are validated with spBLEU, and final submissions average the last 15 checkpoints.Source: §3.3 · p. 3
4 Results
This section compares vocabulary sizes and back-translation methods, then describes iterative training and the final submissions.
4.1 The Role of Vocabularies
- On Small Task 2 devtest, 128K outperformed 256K at 23.14 vs. 21.65 spBLEU; separately, finetuning pretrained Trans_small with 256K improved by 0.58 over 128K Trans_small.The authors characterize 128K as better for training from scratch, while the pretrained model offers more gain.Source: Table 3 · p. 3, §4.1 · p. 3
4.2 Different Back-translation Methods
- Constrained sampling scored best among the three methods, while beam search scored worst; sampling-generated diversity was proposed as a possible explanation.The constrained-sampling result differs from the cited bilingual English-German finding, where unconstrained sampling performed best.Source: §4.2 · p. 4
- After 6M monolingual English sentences, adding more yielded little improvement; the authors suggest English dissimilarity limits transferable knowledge.The English-data analysis uses 80M synthetic sentences, including 6M monolingual English data and the other monolingual data.Source: §4.2 · p. 4, Figure 1 · p. 4
4.3 Final Submissions
- The final setup finetuned a pretrained 1.2B M2M_100 model; Trans_big generated synthetic data because it was too large for the evaluation system.The final submission method used constrained sampling on partial monolingual English data.Source: §4.3 · pp. 4–5
- For Small Task 1, only one trial tested 20M monolingual English sentences plus the remaining monolingual data; further exploration was left to future work.The paper notes that more English data might help this task, whose Central/East European languages are described as more similar to English than Asian languages.Source: §4.3 · p. 5
- The final submissions placed second in both small tasks, and iterative finetuning reached 33.01 spBLEU on Small Task 2 devtest.The iterative process adds synthetic data generated first by Trans_big and then by the second-step Trans_base.Source: Table 4 · p. 3, §4.3 · p. 5
5 Conclusion
The authors report that smaller vocabularies perform better and constrained sampling outperforms the other back-translation methods. Combining these findings with partial high-quality synthetic data, their final models placed second in both small tasks.
5 Conclusion
- A pretrained model with a smaller vocabulary performed better, although the evaluation system’s memory and time limits restricted high-quality synthetic-data generation to a 1.2B model with smaller vocabularies.Source: §5 · p. 5
- Combining three findings with iterative training on partial high-quality synthetic data achieved second place in both small tasks.The findings were constrained sampling performing best, smaller vocabularies performing better, and extensive monolingual English data offering a modest improvement.Source: §5 · p. 6
- Constrained sampling performed best among beam search and unconstrained sampling, while extensive monolingual English data offered a modest improvement.Source: §5 · p. 6