Source-linked AI summary
Edinburgh Neural Machine Translation Systems for WMT 16
Rico Sennrich, Barry Haddow, Alexandra Birch
TL;DR
The paper addresses neural machine translation in the WMT16 news task across four English-centered language pairs. It combines attentional encoder-decoding with synthetic data, dropout, and target-bidirectional reranking, achieving broad gains and tied-best constrained results in 7 of 8 directions.
Problem
The paper evaluates how neural translation systems can improve WMT16 news translation across four English-centered language pairs using limited parallel data supplemented by available monolingual data.
Method
The systems combine attentional encoder-decoder networks and BPE with back-translated monolingual data, pervasive dropout, and right-to-left reranking.
Results
The systems were the tied-best constrained systems for 7 of 8 participating translation directions.
Takeaways & Limitations
Synthetic parallel data improved translation quality across all directions, while dropout and right-to-left reranking produced substantial improvements in their tested settings.
Takeaways & Limitations
Dropout did not improve EN→DE after 10 days of training, although the authors speculate that longer training or larger networks could help.
Abstract
from arXiv · showhide
We participated in the WMT 2016 shared news translation task by building neural translation systems for four language pairs, each trained in both directions: English<->Czech, English<->German, English<->Romanian and English<->Russian. Our systems are based on an attentional encoder-decoder, using BPE subword segmentation for open-vocabulary translation with a fixed vocabulary. We experimented with using automatic back-translations of the monolingual News corpus as additional training data, pervasive dropout, and target-bidirectional models. All reported methods give substantial improvements, and we see improvements of 4.3--11.2 BLEU over our baseline systems. In the human evaluation, our systems were the (tied) best constrained system for 7 out of 8 translation directions in which we participated.
1 Introduction
The paper presents Edinburgh’s WMT16 neural translation systems for four English-centered language pairs, combining attentional encoder-decoding, BPE segmentation, synthetic data, dropout, and target-bidirectional models.
- The systems cover English↔Czech, English↔German, English↔Romanian, and English↔Russian in both translation directions.
- The core architecture is an attentional encoder-decoder with BPE subword segmentation for open-vocabulary translation using a fixed vocabulary.
- The implementation, scripts, synthetic data, and trained models were released as open-source resources.
- Experiments add automatic back-translations of monolingual News data, pervasive dropout, and target-bidirectional models.
2 Baseline System
The baseline uses an attentional neural encoder-decoder with BPE subword units, minibatch training, validation-based stopping, and beam-search decoding.
- The baseline is an attentional encoder-decoder network based on dl4mt, enhanced with ensemble decoding and pervasive dropout.
- Models use fixed minibatch, length, embedding, hidden-layer, optimization, validation, and checkpoint-ensemble settings.The supplied passage specifies minibatches of 80, 500-dimensional embeddings, 1024-unit hidden layers, Adadelta, and BLEU validation.
- Decoding uses beam search with beam size 12, with AmuNMT used for some language pairs as a more efficient alternative.
- BPE learns frequent symbol merges from character sequences and applies the ordered merge list to segment new text.
- BPE represents words as learned subword units, balancing vocabulary size against training-sequence length while supporting in-vocabulary segmentation.
- Source and target data are combined for BPE learning, with 89 500 merge operations learned for each language pair.
3 Experimental Features
The experiments exploit monolingual data through back-translation, apply pervasive dropout to address Romanian overfitting, and use right-to-left reranking to complement left-to-right decoding.
- 3.1 Synthetic Training Data: Target-side monolingual News data are translated into the source language and added as synthetic parallel training data.Back-translation uses a reverse-direction system, such as RO→EN for EN→RO.
- 3.1 Synthetic Training Data: The monolingual data are sampled from News Crawl corpora because they are in-domain with respect to the test set.
- 3.1 Synthetic Training Data: Back-translated data range from 2 million to 10 million sentences across translation directions.
- 3.2 Pervasive Dropout: Dropout is applied to all network layers, including recurrent ones, to mitigate overfitting in English↔Romanian experiments.
- 3.2 Pervasive Dropout: The recurrent dropout method uses one mask across time steps, with additional token-level word dropout.
- 3.2 Pervasive Dropout: In English↔Romanian, full-word dropout is 0.1 and dropout for other layers is 0.2.
- 3.3 Target-bidirectional Translation: Right-to-left models rerank left-to-right n-best lists using complementary target context to make averaged probabilities more robust.The reranking experiments use 50-best lists.
- 3.3 Target-bidirectional Translation: The l-r/r-l approach may be criticized because its gains could reflect ensemble diversity from two independent runs.Experiments cited by the authors report that l-r/r-l reranking is stronger than an ensemble of two independent left-to-right runs.
4 Results
Across the evaluated language pairs, synthetic training data, ensembles, dropout, and target-bidirectional reranking produced substantial BLEU improvements over baseline systems. Results varied by language pair, with especially strong gains for English↔Romanian and English↔Russian.
- English↔German: 3.4–5.7 BLEU improvements came from mixing parallel and synthetic data for English↔German, with ensembles adding 1.3–1.7 BLEU and reranking adding 0.6–1.1 BLEU.The submitted system reranked 50-best left-to-right outputs with a right-to-left ensemble.
- English↔Czech: 4.3–4.9 BLEU improvements over baseline were achieved by the final English↔Czech system, while back-translation alone improved performance by 2.2–2.8 BLEU.Right-to-left reranking improved the ensemble by 0.7–1.0 BLEU.
- English↔Czech: Most Czech→English gains from back-translation arrived with the first batch, although additional back-translated data gradually improved performance.Table 4 continued training from the baseline while increasing the amount of synthetic data.
- English↔Romanian: 4.1–5.1 BLEU improvements came from synthetic training data for English→Romanian.The best single system outperformed the ensemble of the last four checkpoints on development data and was submitted as the primary system.
- English↔Russian: 4–4.4 BLEU improvements from synthetic training data and a further 1.1–1.7 BLEU from ensembles were observed for English↔Russian.Because the alphabets differ, the systems used ISO-9 transliteration to learn shared BPE operations.
5 Shared Task Results
Edinburgh’s WMT16 submissions ranked first or tied first across much of the participating directions. Their models also contributed to other high-ranking systems.
- 7 of 8 translation directions ranked as tied best constrained systems for Edinburgh submissions.This includes EN↔RU and RO→EN in addition to the five directions ranked tied first overall.
- 5 of 8 directions ranked tied first overall: EN↔CS, EN↔DE, and EN→RO.
- The models also appeared in QT21-HimL-SysComb and AMU-UEDIN systems with high rankings for EN→RO, EN→RU, and RU→EN.
- The table reports both automatic BLEU rankings and human rankings for primary submitted systems.Human rankings include anonymous online systems and, for EN↔CS, systems from the tuning task.
6 Conclusion
The systems achieved large translation-quality improvements by combining neural translation with synthetic data and additional model variants. The paper reports gains from back-translated data, pervasive dropout, and right-to-left reranking.
- Synthetic parallel training data from back-translated in-domain monolingual target data improved translation quality for all directions.
- Pervasive dropout on all layers gave substantial improvements for English↔Romanian.
- Right-to-left models improved reranking for English↔German and English→Czech.