Source-linked AI summary
The University of Edinburgh's Neural MT Systems for WMT17
Rico Sennrich, Alexandra Birch, Anna Currey, Ulrich Germann, Barry Haddow, Kenneth Heafield, Antonio Valerio Miceli Barone, Philip Williams
TL;DR
The paper addresses how neural machine translation systems can improve across broad news and biomedical translation tasks through architectural, data, and ensembling choices. It evaluates Nematus systems with deep models, layer normalization, monolingual-data methods, and reranking, finding strong gains across tasks and top constrained-task rankings. The study also identifies narrower caveats, including a backtranslation preprocessing error and instability in one checkpoint ensemble.
Problem
The paper investigates which architecture, monolingual-data, and ensembling choices improve neural translation across WMT17 news and biomedical tasks.
Method
The authors train Nematus neural translation systems with parallel and back-translated data, then evaluate deep architectures, layer normalization, ensembling, and right-to-left reranking through ablation experiments.
Results
The best systems improved news baselines by 2.2–5 BLEU across 12 directions, ranked tied first in 11, and achieved the highest BLEU for every submitted biomedical system.
Takeaways & Limitations
Layer normalization and deep models improved performance across most language pairs, while ensembling and right-to-left reranking also produced gains whose size decreased relative to the previous year.
Takeaways & Limitations
A preprocessing error in Russian-to-English backtranslation made the submitted result worse than the corrected result reported in the paper.
Abstract
from arXiv · showhide
This paper describes the University of Edinburgh's submissions to the WMT17 shared news translation and biomedical translation tasks. We participated in 12 translation directions for news, translating between English and Czech, German, Latvian, Russian, Turkish and Chinese. For the biomedical task we submitted systems for English to Czech, German, Polish and Romanian. Our systems are neural machine translation systems trained with Nematus, an attentional encoder-decoder. We follow our setup from last year and build BPE-based models with parallel and back-translated monolingual training data. Novelties this year include the use of deep architectures, layer normalization, and more compact models due to weight tying and improvements in BPE segmentations. We perform extensive ablative experiments, reporting on the effectivenes of layer normalization, deep architectures, and different ensembling techniques.
1 Introduction
The University of Edinburgh submitted Nematus neural translation systems for broad WMT17 news and biomedical coverage, extending a BPE-based setup with monolingual data and new architectures. Across 12 news directions, the best systems improved substantially over baselines and ranked highly among constrained submissions.
- The systems used Nematus neural machine translation with subword models, back-translated monolingual data, and right-to-left reranking.
- This year’s experiments added deep architectures, new methods for incorporating monolingual data, compact models, layer normalization, and Adam optimization.
- 2.2–5 BLEU separated this year’s baseline models from the best results across all 12 news-task language pairs.
- The submissions ranked tied first for 11 of 12 participating news directions among constrained systems, while all submitted biomedical systems achieved the highest BLEU.
- 1.5–3 BLEU gains over WMT16 single models were observed for six repeated language pairs, with most of the increase attributed to layer normalization and deeper models.
2 Novelties
The paper’s novelties span model architecture, subword processing, training data, and resource efficiency. The experiments show that copied monolingual data can help some directions, while deeper models and compact parameterization make larger systems feasible on single GPUs.
- Subword segmentation and compact models: Rare-subword filtering reduces vocabulary symbols and enables more compact models with only a minor sequence-length increase.For EN↔DE, filtering reduced the English vocabulary from 80581 to 51092 symbols while increasing sequence length by 0.2%.
- Optimization: Layer normalization and Adam were used across systems and were found in preliminary experiments to speed convergence and improve performance.
- Deep architectures: The deep transition architecture replaces a simple GRU recurrence with multiple sequential GRU transitions having independently trainable parameters.The encoder uses a bidirectional recurrent structure, and the decoder extends the baseline transition design to arbitrary depth.
- Deep architectures: The stacked architecture alternates forward and backward GRU layers and uses residual connections between stack levels.
- Monolingual data: Copied target-side monolingual data produced modest improvements or no damage for EN↔TR, leading the authors to adopt it for all EN↔TR experiments.
- Monolingual data: For EN↔LV, copied monolingual data decreased dev-set quality by about 0.5 BLEU but improved newstest2017 quality by 0.1 BLEU.
- Compact models: Weight tying and vocabulary reduction allowed deep models to be trained on single GPUs with 8–12GB memory without model parallelism.
3 System Overview
The systems use constrained, BPE-based neural translation models with Nematus and compare checkpoint-based with independently trained ensembles. Training incorporates weight tying, validation-based model selection, and standard preprocessing choices.
- All systems are constrained to use data from the shared-task website.
- The preprocessing pipeline uses Moses tokenization and truecasing followed by 90000 joint BPE operations, with language-specific variations for Russian and Chinese.
- Nematus implements the systems as attentional encoder-decoder models with 500- or 512-dimensional embeddings and 1024-dimensional hidden layers.
- Target-side embeddings are tied to the transpose of the output weight matrix, while training uses Adam and stops after validation cross-entropy fails to improve for 10 consecutive save-points.
- Checkpoint ensembles reuse the last N checkpoints from one run, whereas independent ensembles train N separate models and can provide greater diversity at higher cost.
4 Experiments
The Chinese and Czech experiments combine large parallel corpora with back-translated data and ensembles of models trained under varied architectures and regimes. Final systems use directional rescoring and reranking to combine complementary models.
- Chinese ↔English: Chinese training adds back-translated monolingual data, yielding approximately 24M parallel sentences plus synthetic data for both translation directions.
- Chinese ↔English: Chinese systems use three runs per target direction, mixing stacked and deep transition architectures with fine-tuned and mixed training regimes.
- Chinese ↔English: The Chinese final system ensembles the best target left-right models, rescoring with target right-left models and reranking.
- Czech ↔English: Czech training uses CzEng 1.6pre and additional WMT corpora, cleaned with language identification, then augments them with about 20M synthetic CS→EN and nearly 6M EN→CS sentences.
- Czech ↔English: Czech systems combine multiple left-right and right-left models, using stacked or deep transition architectures and rescoring or reranking the resulting hypotheses.
4.3 German ↔English
The German↔English systems supplement organizer-provided data with back-translated monolingual news and use deep bidirectional ensembles for hypothesis generation and reranking.
- German↔English training supplements shared-task data with approximately 10 million back-translated monolingual sentences for each direction.
- Eight independent deep models are trained per direction: four left-to-right and four right-to-left.
- The left-to-right ensemble generates 50-best hypotheses, which the right-to-left ensemble reranks.
4.4 Latvian ↔English
The Latvian↔English systems use filtered, language-specific preprocessing and back-translated monolingual data. Final models are selected independently in both directions and combined through length-normalized rescoring.
- EN→LV uses a Latvian-specific tokenizer and filtered parallel data, while LV→EN applies standard preprocessing after filtering the same noisy sentence pairs.
- The systems reserve 2000 sentences for validation and use newsdev2017 as the test set.
- Back-translation uses WMT monolingual English and Latvian news corpora, with phrase-based systems selected because they achieved higher BLEU than preliminary parallel-only neural systems.
- Each direction trains eight independent models, selects checkpoints using newsdev2017, and rescoring uses four-model directional ensembles with 50-best outputs.
4.5 Russian ↔English
The Russian↔English systems combine in-domain back-translated news data with deep-transition models and extensive left-to-right/right-to-left ensembling. A preprocessing error affected the submitted RU→EN result, making it worse than the corrected result reported in the paper.
- Training data: The systems augment parallel data with 7.1M Russian and 20.4M English News Crawl 2016 sentences, mixed 1:1 with human-translated data.Edinburgh’s WMT16 systems produced the backtranslations, and the experiments used the deep transition architecture.
- Ensembling: Final models comprise eight independent systems: four left-to-right and four right-to-left models selected using newsdev2017.A left-to-right ensemble generates 50-best hypotheses, which are rescored with right-to-left models.
- Ensembling: The 50-best output from the left-to-right ensemble is rescored using the right-to-left models.
- Caveat: A preprocessing error in RU→EN backtranslation data caused the submission result to be worse than the corrected result reported in the paper.
4.6 Turkish ↔English
The Turkish↔English systems use parallel data, target-side monolingual data, and a stacked architecture. They compare mixed training with copied and back-translated data, using bidirectional ensembles for reranking.
- Data and preprocessing: About 200k parallel sentences are used for both Turkish↔English directions after preprocessing.Rare subword units are retained because the available training corpus is relatively small.
- Monolingual data: Each direction incorporates about 400k randomly selected WMT News Crawl 2016 sentences as both copied and back-translated target-side data.The backtranslated corpus is created with a shallow NMT system trained on the parallel data.
- Architecture and decoding: All systems use the stacked architecture and eight models per direction, with left-to-right ensembles reranked by right-to-left ensembles.The left-to-right ensemble supplies the 50-best translation hypotheses for reranking.
4.7 Biomedical Task Systems
The biomedical systems adapt generic neural translation models with synthetic in-domain data drawn from NHS 24 and Cochrane websites. Systems differ by language in their generic corpora, adaptation data, architectures, and ensembling, while Romanian additionally requires diacritic normalization.
- Overview: Biomedical systems for English→Czech, German, Polish, and Romanian use task-specific data and models similar to those used for news translation.The provided UFAL corpus is filtered to remove sentence pairs lacking ASCII letters before preprocessing.
- Domain adaptation: Fine-tuning on the relatively small EMEA corpus alone did not give good results, so the systems generated synthetic in-domain data instead.The authors describe EMEA as drug-information leaflets that were not sufficiently close to the target domain.
- Synthetic Data: The synthetic-data pipeline crawls NHS 24 and Cochrane websites, machine-translates the resulting English corpora into four target languages, and applies Moore-Lewis selection.The crawled corpora contain about 64k NHS 24 segments and 174k Cochrane segments.
- Romanian adaptation: Romanian CommonCrawl data has inconsistent diacritics, so a Romanian NMT diacritiser maps stripped text to corrected Romanian before adaptation.The diacritiser is trained on Europarl, DGT, and SETIMES2 data from OPUS.
- Domain adaptation: Adaptation corpora are combined for Romanian, NHS 24 data alone is used for Polish, and separate NHS 24 and Cochrane systems are built for German and Czech.The effect of this domain adaptation is evaluated in Section 5.
- Language-specific systems: English→Czech uses shallow Nematus fine-tuning with 4M sentences per domain, while English→German adds 10M generic sentences and 10k Cochrane abstract pairs.
- Language-specific systems: English→Polish uses 39M generic and 19M in-domain sentence pairs, combining four target left-to-right systems with two reranking systems.The component systems mix fine-tuned or mixed training with stacked or deep-transition architectures.
- Evaluation caveat: Some contrastive or ablative results use converged systems trained after the submission deadline, whereas the reported submission line reflects output available by the deadline.
5 Results
The WMT17 results compare systems across news and biomedical translation, including architecture, ensembling, reranking, and domain-adaptation effects. Performance generally improves over baselines and prior systems, but adaptation and checkpoint ensembling are not uniformly beneficial.
- News translation: 1.5–3 BLEU improvements were observed for single models over WMT16 systems on language pairs evaluated in both years.Some gains were already present in baseline systems and reflect differences in training data, preprocessing, and hyperparameters.
- News translation: Layer normalization and deep models improved BLEU across most language pairs.
- Ensembling and reranking: Checkpoint ensembles generally improved over single models, while independently trained ensembles were consistently more effective.
- Ensembling and reranking: Right-to-left reranking improved scores by 0–1 BLEU this year, below the average 1 BLEU improvement reported in 2016.
- Biomedical translation: Biomedical results compare generic and fine-tuned systems using single best models and ensembles of the last four checkpoints.The comparison uses parallel-only generic training versus fine-tuning with synthetic and in-domain data, including EMEA data.
- Biomedical translation: Domain adaptation improved BLEU on all EN→PL test sets but had mixed effects for EN→RO.For one EN→RO checkpoint ensemble, lower BLEU coincided with outputs about 10% longer because of more nonsense sentences; volatility may affect checkpoint selection by iteration count.
6 Conclusions
The paper presents Edinburgh’s WMT17 neural translation systems and evaluates architecture, data, ensembling, and reranking choices. The systems achieved leading constrained-task results while improving over prior submissions, although reranking gains were smaller than before.
- The paper evaluates Edinburgh’s WMT17 submissions for news and biomedical translation through extensive ablation experiments.
- Layer normalization and deep models improved performance across most language pairs beyond strong systems using parallel and back-translated monolingual data.
- Ensembling and right-to-left reranking improved performance, but their gains decreased slightly compared with the previous year despite independently trained ensembles.
- Edinburgh ranked tied first in 11 of 12 constrained news directions and achieved the highest BLEU for every submitted biomedical system.