Edinburgh Neural Machine Translation Systems for WMT 16

• The systems cover English↔Czech, English↔German, English↔Romanian, and English↔Russian in both translation directions.Source: English↔Russian · p. 1
• The core architecture is an attentional encoder-decoder with BPE subword segmentation for open-vocabulary translation using a fixed vocabulary.Source: English↔Russian · p. 1
• The implementation, scripts, synthetic data, and trained models were released as open-source resources.Source: English↔Russian · p. 1
• Experiments add automatic back-translations of monolingual News data, pervasive dropout, and target-bidirectional models.Source: English↔Russian · p. 1

• The baseline is an attentional encoder-decoder network based on dl4mt, enhanced with ensemble decoding and pervasive dropout.Source: §2 · p. 1
• Models use fixed minibatch, length, embedding, hidden-layer, optimization, validation, and checkpoint-ensemble settings.The supplied passage specifies minibatches of 80, 500-dimensional embeddings, 1024-unit hidden layers, Adadelta, and BLEU validation.Source: §2 · p. 1
• Decoding uses beam search with beam size 12, with AmuNMT used for some language pairs as a more efficient alternative.Source: §2 · p. 1
• BPE learns frequent symbol merges from character sequences and applies the ordered merge list to segment new text.Source: §2.1 · p. 2
• BPE represents words as learned subword units, balancing vocabulary size against training-sequence length while supporting in-vocabulary segmentation.Source: §2.1 · p. 2
• Source and target data are combined for BPE learning, with 89 500 merge operations learned for each language pair.Source: §2.1 · p. 2

3.1 Synthetic Training Data

• Target-side monolingual News data are translated into the source language and added as synthetic parallel training data.Back-translation uses a reverse-direction system, such as RO→EN for EN→RO.Source: §3.1 · p. 2
• The monolingual data are sampled from News Crawl corpora because they are in-domain with respect to the test set.Source: §3.1 · p. 2
• Back-translated data range from 2 million to 10 million sentences across translation directions.Source: §3.1 · p. 2

3.2 Pervasive Dropout

• Dropout is applied to all network layers, including recurrent ones, to mitigate overfitting in English↔Romanian experiments.Source: §3.2 · p. 2
• The recurrent dropout method uses one mask across time steps, with additional token-level word dropout.Source: §3.2 · p. 2
• In English↔Romanian, full-word dropout is 0.1 and dropout for other layers is 0.2.Source: §3.2 · p. 2

3.3 Target-bidirectional Translation

• Right-to-left models rerank left-to-right n-best lists using complementary target context to make averaged probabilities more robust.The reranking experiments use 50-best lists.Source: §3.3 · p. 3
• The l-r/r-l approach may be criticized because its gains could reflect ensemble diversity from two independent runs.Experiments cited by the authors report that l-r/r-l reranking is stronger than an ensemble of two independent left-to-right runs.Source: §3.3 · p. 3

English↔German

• 3.4–5.7 BLEU improvements came from mixing parallel and synthetic data for English↔German, with ensembles adding 1.3–1.7 BLEU and reranking adding 0.6–1.1 BLEU.The submitted system reranked 50-best left-to-right outputs with a right-to-left ensemble.Source: §4.1 · p. 3

English↔Czech

• 4.3–4.9 BLEU improvements over baseline were achieved by the final English↔Czech system, while back-translation alone improved performance by 2.2–2.8 BLEU.Right-to-left reranking improved the ensemble by 0.7–1.0 BLEU.Source: §4.2 · p. 3
• Most Czech→English gains from back-translation arrived with the first batch, although additional back-translated data gradually improved performance.Table 4 continued training from the baseline while increasing the amount of synthetic data.Source: §4.2 · pp. 3–4, Table 4 · p. 4

English↔Romanian

• 4.1–5.1 BLEU improvements came from synthetic training data for English→Romanian.The best single system outperformed the ensemble of the last four checkpoints on development data and was submitted as the primary system.Source: §4.3 · p. 4

English↔Russian

• 4–4.4 BLEU improvements from synthetic training data and a further 1.1–1.7 BLEU from ensembles were observed for English↔Russian.Because the alphabets differ, the systems used ISO-9 transliteration to learn shared BPE operations.Source: §4.4 · p. 4

• 7 of 8 translation directions ranked as tied best constrained systems for Edinburgh submissions.This includes EN↔RU and RO→EN in addition to the five directions ranked tied first overall.Source: §5 · p. 4
• 5 of 8 directions ranked tied first overall: EN↔CS, EN↔DE, and EN→RO.Source: §5 · p. 4
• The models also appeared in QT21-HimL-SysComb and AMU-UEDIN systems with high rankings for EN→RO, EN→RU, and RU→EN.Source: §5 · pp. 4–5
• The table reports both automatic BLEU rankings and human rankings for primary submitted systems.Human rankings include anonymous online systems and, for EN↔CS, systems from the tuning task.Source: Table 7 · p. 5

• Synthetic parallel training data from back-translated in-domain monolingual target data improved translation quality for all directions.Source: §6 · p. 5
• Pervasive dropout on all layers gave substantial improvements for English↔Romanian.Source: §6 · p. 5
• Right-to-left models improved reranking for English↔German and English→Czech.Source: §6 · p. 5