Source-linked AI summary
A neural interlingua for multilingual machine translation
Yichao Lu, Phillip Keung, Faisal Ladhak, Vikas Bhardwaj, Shaonan Zhang, Jason Sun
TL;DR
Multilingual NMT faces limits in scaling bilingual models and in existing approaches to direct zero-shot translation and vocabulary handling. This paper adds a neural interlingua to multilingual encoder-decoder NMT, reporting direct zero-shot translation, crosslingual review classification, and results comparable to bilingual baselines. Its future-work discussion identifies a significant performance gap between direct zero-shot and pivot translation.
Problem
Bilingual models scale quadratically with language count, while existing multilingual approaches have limitations in zero-shot translation and output-vocabulary size.
Source: §1.1 · pp. 1–2Method
The paper incorporates an explicit neural interlingua into multilingual NMT to learn language-independent sentence representations.
Source: §1.2 · p. 2, §5 · p. 8Results
The model performs direct zero-shot translation, supports French and German review classification from an English Yelp classifier, and yields results comparable to bilingual baselines.
Source: §5 · p. 8Takeaways & Limitations
The learned representations are language-independent and support zero-shot translation and crosslingual text classification.
Source: §5 · p. 8The authors identify a significant performance gap between direct neural zero-shot translation and pivot translation as future work.
Source: §5 · pp. 8–9
Abstract
from arXiv · showhide
We incorporate an explicit neural interlingua into a multilingual encoder-decoder neural machine translation (NMT) architecture. We demonstrate that our model learns a language-independent representation by performing direct zero-shot translation (without using pivot translation), and by using the source sentence embeddings to create an English Yelp review classifier that, through the mediation of the neural interlingua, can also classify French and German reviews. Furthermore, we show that, despite using a smaller number of parameters than a pairwise collection of bilingual NMT models, our approach produces comparable BLEU scores for each language pair in WMT15.
1 Introduction
Multilingual NMT can reduce the quadratic scaling of bilingual models, but existing approaches have limitations in direct zero-shot translation and vocabulary handling. This work introduces a neural interlingua and demonstrates direct zero-shot translation and crosslingual review classification.
1.1 Multilingual Machine Translation
- Training separate bilingual models scales quadratically with the number of languages, while multilingual approaches combine language-specific encoders and decoders.Prior approaches include one-to-many, many-to-one, and many-to-many translation settings.Source: §1.1 · p. 1
- Existing multilingual models have limitations: direct zero-shot translation was impossible in one approach, while a universal model’s output vocabulary can be costly to enlarge.Training and inference time scale linearly with the size of the decoder’s output layer.Source: §1.1 · pp. 1–2
1.2 Our Contributions
- The neural interlingua maps language-specific encoder embeddings to representations agnostic to source and target languages.It is introduced as an explicit component of the multilingual NMT architecture.Source: §1.2 · p. 2
- Direct zero-shot translation improves on the universal encoder-decoder results reported in Miura et al. (2017) for Fr↔Ru, Zh↔Es, and Es↔Fr.The translations do not pivot through an intermediate language.Source: §1.2 · p. 2
- Source sentence embeddings support an English Yelp classifier that also classifies French and German reviews through the interlingua.Parallel translations’ sentence embeddings are also close to each other in a low-dimensional space.Source: §1.2 · p. 2
2 Model Architecture
The model routes source sentences through language-specific encoders, a shared neural interlingua, and language-specific decoders. The interlingua supplies a common representation, while the architecture’s encoder, decoder, and attention counts scale linearly with the number of languages.
2 Model Architecture
- Language-specific encoders pass their embeddings through a shared recurrent interlingua before language-specific decoders generate translations.The interlingua maps language-specific representations to an interlingual representation, and decoders receive the source only through that representation.Source: §2 · pp. 2–3, Figure 1 · p. 2
- Each minibatch contains one source and one target language, and only their encoder, the interlingua, and their decoder participate in its passes.Source-target language pairs rotate according to a schedule during training.Source: §2 · p. 2
- The interlingua output has a fixed length of 50 in the experiments, matching the maximum source sentence length used in training.The model omits source- and target-language indicator tokens.Source: §2 · p. 3
- The number of encoders, decoders, and attention mechanisms scales linearly with the number of languages.A common source representation from the interlingua is available to all decoders.Source: §2 · p. 3
- The neural interlingua is independent of the chosen architecture, though the experiments use an LSTM encoder-decoder with single-headed attention.The paper notes that the component could also be introduced to transformer or CNN encoder-decoder networks.Source: §2 · p. 3
3 Experiments
Across four experiments, the multilingual model achieved broadly similar WMT15 BLEU scores to bilingual baselines, improved direct zero-shot translation over the universal encoder-decoder approach, and supported crosslingual Yelp classification and interlingua visualization.
3.1 Model Training
- The multilingual model uses one bidirectional LSTM encoder layer per input language, an attentional LSTM interlingua layer, and one attentional LSTM decoder layer per output language.Bilingual baselines use two bidirectional LSTM encoder layers; training used WMT15 English-parallel data and importance sampling.Source: §3.1 · p. 4, §3.3 · p. 5
3.3 Multilingual NMT versus Bilingual NMT
- Generally <1.0 BLEU separates the multilingual model and bilingual baselines across newstest2015 and newsdiscuss2015, with lower-resource gains and higher-resource decreases.The authors suspect the language-pair schedule, which cycles through pairs as though they were equally frequent, contributes to this pattern.Source: Table 2 · p. 5, §3.3 · p. 5
3.2 Language Rotation During Training
- Without identity language pairs, Fr-Ru zero-shot BLEU was <1.0; adding identity pairs encourages source embeddings to work with an additional decoder.The initial schedule failed to produce useful zero-shot translation or crosslingual classification representations because WMT15 is not multi-parallel.Source: §3.2 · pp. 4–5
3.3 Multilingual NMT versus Bilingual NMT
- Copied monolingual data did not improve baseline BLEU; even WMT15’s smallest language pair had 2 million sentence pairs, over three times the cited low-resource pairs.The model generally improved when translating to English, which the authors associate with the English decoder seeing more English text.Source: §3.3 · p. 5
3.4 Zero-shot Multilingual Classification
- An English-trained logistic-regression classifier using mean-pooled 512-dimensional interlingual embeddings achieved accuracy similar to classification using English-translated French and German reviews.The classifier was trained on English reviews and applied to French and German review embeddings.Source: §3.4 · pp. 5–6
3.5 Direct Zero-shot Translation
- On Es-Fr and Fr-Es, the model is within <2.0 BLEU of pivot translation while generally trailing pivot and outperforming the universal encoder-decoder approach.The UN evaluation tested three language pairs absent from training, with 4,000 test sentence pairs per language pair; the authors note that parity with pivot translation could reduce latency and computational overhead.Source: §3.5 · p. 7, Table 5 · p. 7
3.6 Interlingua Visualization
- In the visualization, parallel English, French, and Russian sentence embeddings form four groups, with within-group points close together and different groups separated.The 512-dimensional mean-pooled embeddings were projected to two dimensions with PCA.Source: §3.6 · p. 7, Figure 3 · p. 7
4 Related Work
Prior multilingual translation systems combine language-specific encoders and decoders in several ways, or use a universal encoder-decoder. The related work contrasts their attention, vocabulary, and zero-shot translation trade-offs with this paper’s interlingua approach.
4.1 Networks with Language-specific Encoders and Decoders
- Earlier multilingual designs include two-encoder many-to-one and one-encoder/multiple-decoder systems; the former requires two source sentences per target at prediction.The two-encoder approach primarily considered the trilingual case with a multi-parallel corpus.Source: §4.1 · pp. 7–8
- One-to-many improved translation through increased encoder exposure and multitask learning, while shared attention avoids quadratic scaling across language pairs.The shared-attention model uses language-specific encoders and decoders but shares their attention mechanisms.Source: §4.1 · p. 8
- Because attention aligns source and target, one shared mechanism may restrict divergent word orders; this approach assigns one attention mechanism per decoder.Source: §4.1 · p. 8
4.2 Universal Encoder-Decoder Networks
- Shared vocabularies may force a large, slow output layer or a constrained layer that hurts translation; this approach keeps target-specific decoders without vocabulary-growth speed costs.Source: §4.2 · p. 8
4.3 Zero-shot Translation
- Zero-shot translation addresses missing parallel text; prior methods use pivot languages, while universal encoder-decoders perform direct translation but can lose BLEU on some pairs.Source: §4.3 · p. 8
5 Conclusion
The paper concludes that its neural interlingua supports direct zero-shot translation and crosslingual review classification. Results improve on reported universal encoder-decoder zero-shot performance, while the authors identify a remaining gap versus pivot translation.
5 Conclusion
- The model incorporates an explicit neural interlingua into multilingual NMT and learns language-independent sentence representations.Source: §5 · p. 8
- Direct zero-shot translation across three pairs improved over reported universal encoder-decoder results; English Yelp embeddings also classified German and French reviews, with results comparable to bilingual baselines.Source: §5 · p. 8
- The authors identify a significant direct-zero-shot versus pivot-translation performance gap and plan to manipulate sentence embeddings to improve results.Source: §5 · pp. 8–9