Source-linked AI summary
Multilingual Neural Machine Translation with Knowledge Distillation
Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, Tie-Yan Liu
TL;DR
Multilingual NMT is efficient but usually less accurate than separate models as language diversity and the number of language pairs increase. The paper trains one multilingual student with both ground-truth translations and outputs from individual language-pair teachers. Across three datasets and up to 44 languages, the method nearly matches or sometimes outperforms individual models using only 1/N of their parameters.
Problem
With dozens of language pairs, multilingual NMT usually achieves inferior accuracy to individual models, despite its single-model efficiency.
Method
Individual language-pair models serve as teachers, while one multilingual model learns simultaneously from their outputs and ground-truth translations.
Results
Experiments on three datasets with up to 44 languages show that the multilingual model nearly matches or outperforms individual models with only 1/N of their parameters.
Takeaways & Limitations
One multilingual model can handle up to 44 languages with comparable or better accuracy than separate individual models in these experiments.
Takeaways & Limitations
The authors leave the method’s behavior on larger datasets and hundreds or thousands of language pairs for future study.
Abstract
from arXiv · showhide
Multilingual machine translation, which translates multiple languages with a single model, has attracted much attention due to its efficiency of offline training and online serving. However, traditional multilingual translation usually yields inferior accuracy compared with the counterpart using individual models for each language pair, due to language diversity and model capacity limitations. In this paper, we propose a distillation-based approach to boost the accuracy of multilingual machine translation. Specifically, individual models are first trained and regarded as teachers, and then the multilingual model is trained to fit the training data and match the outputs of individual models simultaneously through knowledge distillation. Experiments on IWSLT, WMT and Ted talk translation datasets demonstrate the effectiveness of our method. Particularly, we show that one model is enough to handle multiple languages (up to 44 languages in our experiment), with comparable or even better accuracy than individual models.
1 INTRODUCTION
Multilingual NMT reduces the resource cost of serving many language pairs in one model, but language diversity makes its accuracy inferior to separate individual models as the number of pairs grows. The paper addresses this gap by distilling knowledge from individual models into a multilingual model.
- Multilingual NMT handles multiple language pairs in one model, reducing offline training and online serving costs.
- When multilingual NMT covers dozens or hundreds of language pairs, language diversity usually lowers accuracy below that of individual models.
- Training a multilingual model with accuracy comparable to individual models remains challenging for dozens of language pairs.
- The proposed method trains individual language-pair models as teachers, then trains one multilingual student to match their outputs and the ground-truth translations.
- Experiments cover IWSLT, WMT, and Ted talk datasets, with 44 language pairs in the largest setting.
2 BACKGROUND
Neural machine translation models conditional token generation, while multilingual NMT extends one model across many language pairs. Prior work often trades accuracy against scale, motivating multilingual distillation with multiple teachers and one student.
- An encoder-decoder NMT model maps an input sentence to hidden representations and generates target tokens conditioned on prior tokens and those representations.
- Multilingual NMT shares model components across multiple language pairs, including encoders, decoders, attention modules, or an entire universal model.
- With dozens of language pairs, multilingual NMT usually has lower accuracy than separate individual models.
- Knowledge distillation was initially used to compress large teacher models into compact students while preserving accuracy.
- This paper uses multiple teacher models to train a single multilingual student, unlike codistillation methods that collaboratively train multiple students.
3 METHOD
The method combines ground-truth training with teacher-output matching so one multilingual student can learn from separate language-pair teachers. Selective distillation disables teacher matching when the student sufficiently surpasses a teacher, while top-K storage reduces memory demands.
- The multilingual model is taught by individual models because separate models usually achieve higher accuracy when many language pairs have sufficient data.
- Knowledge distillation adds cross-entropy matching to the teacher’s output distribution alongside the standard one-hot-label loss.
- The total loss interpolates negative log-likelihood and distillation losses using coefficient λ.
- For each language pair, the multilingual student uses the corresponding individual teacher’s loss together with the original log-likelihood loss.
- Selective distillation removes the distillation loss after the student exceeds a teacher by threshold τ, but can re-enable it if the student later falls behind.
- Teacher probability distributions are generated offline and only top-K probabilities are loaded and normalized to reduce GPU memory costs.
4 EXPERIMENTS
Experiments across IWSLT, WMT, and Ted talk evaluate multilingual translation with knowledge distillation against individual and baseline multilingual models. The method consistently improves multilingual baselines and often matches or exceeds individual-model accuracy, including with 44 languages.
- Settings: Experiments cover 12 IWSLT, 6 WMT, and 44 Ted talk language pairs using multilingual and individual translation models.The study reports results for many-to-one and one-to-many settings where applicable.
- IWSLT results: On IWSLT many-to-one translation, multi-distillation improves over the multilingual baseline for all languages and matches or exceeds individual models on 10 of 12 languages.The distilled model uses only 1/12 of the summed individual-model parameters.
- IWSLT results: In the IWSLT one-to-many setting, multi-distillation maintains or improves accuracy versus individual models and raises the multilingual baseline by nearly 1 BLEU score.The paper describes one-to-many translation as more difficult because it involves different target languages.
- WMT results: On WMT, the multilingual baseline trails individual models on 5 of 6 languages, while multi-distillation outperforms individual models on all 6 and improves some by more than 2 BLEU scores.For one-to-many WMT translation, the method exceeds the multilingual baseline by more than 1 BLEU score on nearly all languages.
- Ted talk results: On Ted talk translation across 44 languages, multi-distillation improves over the multilingual baseline for every language, usually by more than 1 BLEU score, while using 1/44 of the summed individual-model parameters.It matches or surpasses individual models for most languages, with larger gains correlated with smaller language-data sizes.
- Analysis: Selective distillation performs better than distillation throughout training on 13 of 16 analyzed languages, while increasing top-K outputs beyond 8 yields no further accuracy gains.Back distillation improves 9 of 12 individual language models, and word-level distillation consistently outperforms sequence-level distillation on IWSLT.
5 CONCLUSION
The paper proposes knowledge distillation to improve multilingual NMT, which is typically less accurate than separate individual models. Across three datasets with up to 44 languages, the method nearly matches or surpasses individual models using 1/N model parameters.
- Knowledge distillation transfers knowledge from individual translation models to a single multilingual model to boost its accuracy.The individual models serve as teachers, while the multilingual model serves as the student.
- Experiments on three translation datasets with up to 44 languages show that the distilled multilingual model can nearly match or outperform individual models.
- 1/N model parameters are sufficient for the distilled multilingual model, with N reaching 44 in the experiments.
1 DATASET DESCRIPTION
The experiments use IWSLT, WMT, and Ted Talk translation datasets covering 12, 6, and 44 language pairs or languages, respectively. Official validation and test sets are used, with training data sizes documented in accompanying tables.
- Three datasets are used: IWSLT with 12 languages↔English pairs, WMT with 6 languages↔English pairs, and Ted Talk with 44 selected languages.The Ted Talk languages were selected because they had sufficient data for the experiments.
- IWSLT data come from evaluation campaigns from 2014 to 2016, with roughly 80K to 200K sentence pairs per language pair.Official validation and test sets are used for each language pair.
- WMT includes five language↔English pairs from the 2016 dataset and Latvian↔English from the 2017 dataset.Official validation and test sets are used for each language pair.
- The Ted Talk corpus contains multilingual translations, and 44 languages with sufficient data are selected for the experiments.Official validation and test sets are used for each language pair.
2 LANGUAGE NAME AND CODE
The paper lists language names and their ISO 639-1 codes in Table 13. Two dataset-specific codes represent Brazilian Portuguese and Canadian French.
- Language names and corresponding codes are listed according to the ISO 639-1 standard.
- Ptbr denotes Portuguese spoken in Brazil, while Frca denotes French spoken in Canada.
3 RESULTS ON TED TALK DATASET
On the 44-language Ted Talk translation task, the multilingual baseline underperforms individual models, whereas the proposed distilled multilingual model nearly matches or surpasses them. The single model therefore reduces parameters to 1/44 without loss of accuracy.
- 44-language→English BLEU scores for individual and multilingual models are reported in Table 14.
- The multilingual baseline performs worse than the individual model, while the proposed multilingual model nearly matches or outperforms it.
- 1/44 model parameters can handle all 44 languages without loss of accuracy.