Source-linked AI summary
Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, Jeffrey Dean
TL;DR
The paper tackles the difficulty of translating multiple language pairs with a single efficient NMT system. It uses a shared encoder-decoder model with a target-language token and reports improved low-resource translation, multilingual performance, and zero-shot translation, while noting quality degradation for unrelated language pairs and lower-quality separated zero-shot representations.
Problem
Existing NMT systems were generally built for single language pairs, lacking a sufficiently simple and efficient single-model solution for multiple pairs.
Method
The method shares all model parameters across multilingual data and adds an artificial input token specifying the target language, with mixed-language batches during training.
Results
The multilingual system improves low-resource translation and enables zero-shot translation between unseen language pairs, including transfer without explicit bridging.
Takeaways & Limitations
A single shared model can support many language pairs while improving some trained pairs and enabling transfer to language combinations absent from training.
Takeaways & Limitations
Zero-shot translation performs worse than explicit bridging for Spanish→Japanese, with roughly a 50% BLEU-score drop, and separated zero-shot translations tend to have lower BLEU scores.
Abstract
from arXiv · showhide
We propose a simple solution to use a single Neural Machine Translation (NMT) model to translate between multiple languages. Our solution requires no change in the model architecture from our base system but instead introduces an artificial token at the beginning of the input sentence to specify the required target language. The rest of the model, which includes encoder, decoder and attention, remains unchanged and is shared across all languages. Using a shared wordpiece vocabulary, our approach enables Multilingual NMT using a single model without any increase in parameters, which is significantly simpler than previous proposals for Multilingual NMT. Our method often improves the translation quality of all involved language pairs, even while keeping the total number of model parameters constant. On the WMT'14 benchmarks, a single multilingual model achieves comparable performance for English$\rightarrow$French and surpasses state-of-the-art results for English$\rightarrow$German. Similarly, a single multilingual model surpasses state-of-the-art results for French$\rightarrow$English and German$\rightarrow$English on WMT'14 and WMT'15 benchmarks respectively. On production corpora, multilingual models of up to twelve language pairs allow for better translation of many individual pairs. In addition to improving the translation quality of language pairs that the model was trained with, our models can also learn to perform implicit bridging between language pairs never seen explicitly during training, showing that transfer learning and zero-shot translation is possible for neural translation. Finally, we show analyses that hints at a universal interlingua representation in our models and show some interesting examples when mixing languages.
1 Introduction
The paper addresses the difficulty of supporting multiple language pairs with one efficient NMT model. It proposes a shared multilingual system whose benefits include simpler deployment, improved low-resource translation, and zero-shot transfer.
- A single multilingual model addresses the lack of a simple, efficient way to handle multiple language pairs without major NMT architecture changes.
- Simplicity: The method simplifies scaling and deployment by adding multilingual data and reducing the number of separate models needed for production.Batching requests across source and target languages can also improve serving efficiency.
- Low-resource language improvements: Shared parameters significantly improve translation quality for low-resource language pairs when their data are mixed with abundant-resource pairs.
- Zero-shot translation: The model can translate language pairs never seen together during training, demonstrating zero-shot transfer within neural translation.For example, Portuguese→English and English→Spanish training can support Portuguese→Spanish translation.
- The paper evaluates multilingual models on WMT benchmarks, production datasets, transfer-learning experiments, and analyses of implicit bridging and shared representations.
2 Related Work
Earlier multilingual translation systems commonly modified NMT with language-specific components or additional processing. This paper instead presents single-model multilingual and zero-shot translation without extra bridging steps.
- Prior multilingual NMT proposals used separate encoders, decoders, or attention mechanisms for different languages.
- Other approaches shared some components but remained limited to particular source or target-language configurations.Examples include a shared character-level encoder restricted to one target language and multi-source systems with language-specific modules.
- The paper relates its approach to multitask learning and earlier uses of special source tokens for controlling target properties such as politeness or tense.
- Zero-shot translation requires no explicit examples of the tested language pair, whereas zero-resource approaches additionally fine-tune on pseudo-parallel data.
- The authors claim the first validation of true multilingual translation with one encoder-decoder model and the first demonstration of zero-shot transfer without additional steps.
3 System Architecture for Multilingual Translation
The architecture retains the GNMT encoder-decoder system and adds only a target-language token to each input. Shared multilingual training uses a common wordpiece vocabulary while leaving the source language implicit.
- The multilingual architecture is the GNMT architecture, with optional direct encoder-decoder connections used in some experiments.The paper later reports that the effect of these connections is negligible, though trained connections must remain available at inference.
- An artificial token at the beginning of each input specifies the desired target language, such as <2es> for Spanish.The source sentence is reversed as a processing step in the illustrated example.
- The model does not receive an explicit source-language token; it learns the source language from context and can therefore handle code-switching.The trade-off is potential ambiguity for words with identical spellings but different meanings across source languages.
- Multilingual training combines language-pair data and uses a shared wordpiece model across source and target data to limit vocabulary and address unknown words.
- Training and inference require only one additional token per source sentence to specify the target language.
4 Experiments and Results
Experiments show that multilingual NMT can combine language pairs in one shared model with comparable or improved translation quality, while reducing training and production complexity. The shared architecture also supports zero-shot translation and incremental improvement with limited direct parallel data.
- One-to-Many: +0.9 BLEU is the gain reported for English→Spanish, although multilingual models do not consistently outperform baselines when translating into multiple target languages.The decoder may face greater difficulty when combining target languages or scripts in one shared vocabulary.
- Many-to-Many: Approximately 2.5% is the average relative BLEU loss for same-size multilingual production models across many-to-many experiments.Oversampling helps smaller language pairs at the expense of larger ones, while not oversampling reverses that trade-off.
- Large-scale Experiments: One 255M-parameter multilingual model can approach the quality of 12 single-language models totaling 3B parameters.The multilingual model requires roughly one twelfth of the training time and computing resources to converge, though quality varies with model size.
- Zero-Shot Translation: Reasonable Portuguese→Spanish zero-shot translation emerges from models trained without that language pair, and four-pair training improves it by close to 3 BLEU points over two-pair training.The shared architecture is hypothesized to learn an interlingua enabling this transfer.
5 Visual Analysis
The visual analyses examine whether multilingual NMT develops shared semantic representations across languages and whether embedding geometry relates to zero-shot translation quality. Results show semantic clustering in some models, but also separated regions associated with lower-quality zero-shot outputs.
- Shared representations: t-SNE clusters generally group translations of the same underlying sentence across different source and target languages.The analysis used 74 semantically identical English, Japanese, and Korean triples, producing 9,978 context vectors.
- Shared representations: Some multilingual models show strong visual evidence of shared representations, whereas others contain large language-pair-specific clusters.The paper explicitly notes that clean semantic clustering is not universal across models.
- Zero-shot geometry: Zero-shot Portuguese→Spanish translations occupy both a large separated region and regions near embeddings from other languages.The separated region primarily contains zero-shot translations, while some zero-shot vectors fall near other-language embeddings.
- Zero-shot geometry: Zero-shot translations in the separated region tend to have lower BLEU scores, although the paper says this relationship requires further investigation.The authors treat the interpretation of the separated cluster as provisional rather than definitive.
- Embedding-quality relationship: The embedding dissimilarity and zero-shot BLEU scores have a Pearson correlation coefficient of −0.42, indicating moderate correlation.Dissimilarity compares the average pointwise distance between a zero-shot translation and the corresponding translation from a trained language pair.
6 Mixing Languages
The paper tests multilingual models with mixed-language inputs and mixed target-language signals. Code-switched inputs can often translate correctly without code-switched training data, while mixed target signals usually trigger language switching rather than an intermediate language.
- Source language code-switching: Mixed Japanese/Korean source input can produce correct English translations even without code-switched training examples.The model handled different scripts and produced an English translation for the mixed input example.
- Source language code-switching: Some code-switched inputs fail, with the model sometimes copying source-language segments instead of translating them.The paper presents this as a practical limitation of source-side code-switching.
- Target language mixing: Weighted combinations of Japanese and Korean target tokens usually switch from one language to the other around w = 0.5 rather than producing an intermediate language.For intermediate weights, the model can also switch languages mid-sentence.
- Target language mixing: Examples show transitions among Russian, Ukrainian, Belarusian, Japanese, Korean, Spanish, and Portuguese as target-language weights change.In one Japanese/Korean example, increasing the Korean weight eventually yields a more natural Korean sentence.
7 Conclusion
The paper presents a single multilingual NMT model with shared parameters that supports multiple languages and enables zero-shot translation. It also reports low-resource improvements and evidence of an interlingua-like representation.
- A single multilingual NMT model translates between multiple languages while sharing all parameters.
- Zero-shot translation between language pairs unseen in training is possible without explicit bridging.
- Mixing low-resource and abundant language pairs improves translation quality for the low-resource pairs.
- Visual analyses suggest that the models learn an interlingua representation across the involved language pairs.
- The simple architecture permits mixing languages on the source or target side, producing interesting translation examples.