Source-linked AI summary
Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism
Orhan Firat, Kyunghyun Cho, Yoshua Bengio
TL;DR
The paper asks whether multilingual neural translation can share attention across language pairs rather than using independent pair-specific mechanisms. It proposes a multi-way, multilingual attention-based encoder-decoder with multiple encoders and decoders, and reports better performance than single-pair models, especially for low-resource language pairs.
Problem
Existing translation mappings and attention mechanisms are language-pair-specific, creating a challenge for multilingual systems that avoid independently parameterized attention mechanisms.
Method
The model uses multiple source-language encoders and target-language decoders with one attention mechanism shared across all language pairs.
Results
The multilingual model outperforms sets of single-pair models in full and controlled experiments, with especially clear improvements for low-resource language pairs.
Takeaways & Limitations
A single attention-based neural network can handle multiple source and target languages simultaneously, including the WMT’15 language set.
Takeaways & Limitations
The authors identify translation to English as showing larger improvements and leave the explanation and further technique combinations for future research.
Abstract
from arXiv · showhide
We propose multi-way, multilingual neural machine translation. The proposed approach enables a single neural translation model to translate between multiple languages, with a number of parameters that grows only linearly with the number of languages. This is made possible by having a single attention mechanism that is shared across all language pairs. We train the proposed multi-way, multilingual model on ten language pairs from WMT'15 simultaneously and observe clear performance improvements over models trained on only one language pair. In particular, we observe that the proposed model significantly improves the translation quality of low-resource language pairs.
1 Introduction
The paper proposes a multilingual neural translation model that shares one attention mechanism across language pairs, addressing the difficulty of extending pair-specific translation systems. Experiments show better generalization than single-pair models, especially for low-resource translation, and demonstrate multi-way translation in one network.
- Existing machine translation mappings are specific to individual language pairs, making extension to multiple pairs nontrivial.
- The proposed model uses an attention-based encoder-decoder network with multiple encoders and decoders but a shared attention mechanism across language pairs.
- For low-resource language pairs, the multilingual model generalizes better than a single-pair model when parallel data are scarce.
- The improvement for low-resource pairs is not explained only by increased target-side monolingual data.
- A single attention-based network can perform multi-way translation across English, French, Czech, German, Russian, and Finnish.
2 Background: Attention-based Neural Machine Translation
Attention-based neural machine translation represents source sentences as position-dependent context vectors and lets a recurrent decoder select relevant source information while generating each target symbol.
- Attention avoids compressing an arbitrarily long source sentence into one fixed-dimensional vector.
- A bidirectional recurrent encoder constructs a variable-size set of context vectors from the source sentence.
- At each decoding step, attention scores measure the relevance of each context vector for deciding the next target symbol.
- The decoder computes a time-dependent context vector from attention weights, its previous hidden state, the previous decoded symbol, and the full context set.
- The model computes next-symbol probabilities with a parametric output function and trains the differentiable network by maximizing conditional log-likelihood.
3 Multi-Way, Multilingual Translation
The paper extends multilingual neural translation to multiple source and target languages by asking whether attention can be shared across language pairs. Its central challenge is avoiding language-pair-specific attention mechanisms that grow quadratically and limit transfer to resource-poor languages.
- Attention is difficult to share because alignment is largely specific to a language pair.
- A naive design uses O(L^2) separate attention mechanisms, causing quadratic parameter growth and reducing the potential for multilingual transfer.
- The paper frames the key questions as whether one attention mechanism can serve multiple language pairs and how to construct such a model.
4 Multi-Way, Multilingual Model
The proposed architecture uses one encoder per source language, one decoder per target language, and a single attention mechanism shared across all language pairs. Transformations align language-specific encoder and decoder representations with this shared mechanism.
- The model consists of N encoders, M decoders, and a shared attention mechanism for multilingual attention-based translation.
- Encoders: Each source language has an encoder, which can use language-specific architectures before projecting context vectors into a common dimensional space.
- Encoders: Encoder-side transformations make context vectors compatible with the shared attention mechanism.
- Decoders: Each target-language decoder transforms its hidden state and previously decoded symbol to produce inputs compatible with shared attention.
- Attention Mechanism: The shared attention mechanism computes relevance scores, normalizes them into attention weights, and forms context vectors as weighted sums.
- Table 2 reports BLEU scores for target-pair parallel corpora constrained to 5%, 10%, 20%, and 40% of their original sizes.
5 Experiment Settings
The experiments evaluate the model on WMT’15 language pairs using multilingual and low-resource settings, with BLEU and average log-probability as the main metrics. Training uses shared architecture components, bilingual-corpus scheduling, and GPU parallelism for larger models.
- Evaluation: The evaluation covers ten directed WMT’15 pairs between English and French, Czech, German, Russian, and Finnish.
- Data Preprocessing: Training data are tokenized, cleaned, and represented with byte-pair-encoded subwords using 30k symbols per language.
- Evaluation Metric: BLEU is the primary metric, supplemented by average log-probability to assess density estimation without approximate-decoding error.
- Low-Resource Translation: Low-resource experiments retain 5%, 10%, 20%, or 40% of each pair’s original parallel corpus and compare Multi with Single and Single+DF baselines.
- Large-scale Translation: The large-scale model uses five encoders and five decoders trained on the full WMT’15 corpora.
- Model Architecture: The architecture uses 620-dimensional symbol embeddings, bidirectional 1,000-unit-per-direction GRU encoders, and 1,000-unit GRU decoders.
- Training: Language-pair updates are cycled to reduce bias toward any one bilingual corpus, while model paths can be distributed across GPUs.
6 Results and Analysis
The proposed multilingual model improves or matches single-pair systems across most evaluated directions, with especially clear gains for low-resource translation and foreign-language-to-English translation.
- Low-Resource Translation: The Multi model outperforms Single and Single+DF in every low-resource case, with larger gaps as target parallel data decreases.The gains therefore exceed the benefit attributed to adding target-side monolingual data alone.
- Large-Scale Translation: For most of the ten evaluated language-pair directions, the multilingual model outperforms or matches single-pair models on both BLEU and average log-probability.This occurs despite the single-pair model set having twice as many parameters overall.
- Large-Scale Translation: The benefit is more apparent when translating from a foreign language to English.The authors conjecture that English appears more often in the parallel corpora, improving estimation of the English decoder, and propose pseudo-parallel data as a future test.
7 Conclusion
The paper proposes a multi-way, multilingual attention-based neural translation model and evaluates it across WMT’15 languages and controlled low-resource settings. It reports improvements over separate single-pair models, especially for low-resource language pairs, while identifying translation to English as a setting for larger gains and further investigation.
- 7 Conclusion: The proposed model uses one neural network to handle multiple source and target languages simultaneously with attention-based multilingual translation.It extends prior multilingual systems by supporting attention and multi-way translation, and is evaluated with WMT’15 parallel corpora.
- 7 Conclusion: Across full-corpus and controlled-data experiments, the multilingual model improves over a set of single-pair models, especially for low-resource language pairs.The evaluation uses all five WMT’15 languages and artificially controlled target-parallel-corpus sizes.
- 7 Conclusion: Larger improvements were observed when translating to English, which the authors conjecture may reflect greater English availability in the parallel corpora.They identify testing this phenomenon, combining additional translation techniques, and translating unseen language pairs as future work.