Source-linked AI summary
Facebook FAIR's WMT19 News Translation Task Submission
Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, Sergey Edunov
TL;DR
The paper addresses how to improve neural translation for WMT19 across four English–German and English–Russian directions using noisy web data and monolingual data. It combines filtering, back-translation, domain-specific fine-tuning, ensembling, and noisy-channel reranking. Facebook FAIR ranks first in all four human-evaluation directions, with a 4.5 BLEU improvement over its WMT18 En→De submission.
Problem
The submission investigates improving translation quality in four high-resource WMT19 directions while using noisy crawled data and available monolingual resources.
Method
The systems combine bitext filtering, large-scale back-translation, domain-specific fine-tuning, ensembling, and noisy-channel model reranking.
Results
The submissions rank first in all four human-evaluation directions, and En→De improves over WMT18 by 4.5 BLEU.
Takeaways & Limitations
The submission demonstrates strong performance across all four evaluated directions, including En→De performance significantly above other systems and human translations.
Abstract
from arXiv · showhide
This paper describes Facebook FAIR's submission to the WMT19 shared news translation task. We participate in two language pairs and four language directions, English <-> German and English <-> Russian. Following our submission from last year, our baseline systems are large BPE-based transformer models trained with the Fairseq sequence modeling toolkit which rely on sampled back-translations. This year we experiment with different bitext data filtering schemes, as well as with adding filtered back-translated data. We also ensemble and fine-tune our models on domain-specific data, then decode using noisy channel model reranking. Our submissions are ranked first in all four directions of the human evaluation campaign. On En->De, our system significantly outperforms other systems as well as human translations. This system improves upon our WMT'18 submission by 4.5 BLEU points.
1 Introduction
Facebook FAIR extends its WMT19 translation submission to four English–German and English–Russian directions, combining filtered data, back-translation, fine-tuning, ensembling, and noisy-channel reranking. The En→De system improves over WMT18 by 4.5 BLEU.
- Scope and approach: The submission covers En→De, De→En, En→Ru, and Ru→En using subword models, large-scale back-translation, model ensembling, and FAIRSEQ.The systems follow techniques from the previous submission and use the FAIRSEQ sequence modeling toolkit.
- Data and training: High-quality monolingual data remains important in these high-resource directions, so the systems back-translate Newscrawl and selected Commoncrawl data.Commoncrawl is larger and noisier than Newscrawl, motivating filtering and selective use.
- Data and training: The final models use domain-specific fine-tuning and noisy-channel model reranking.These procedures are applied after training on bitext and back-translated data.
- Results: 4.5 BLEU improvement over the WMT18 En→De submission is attributed mainly to larger models, larger-scale back-translation, and noisy-channel reranking.The authors also note that differences in dataset quality account for some gains.
2 Data
The data pipeline combines broad bitext and monolingual resources with filtering, subword segmentation, and selective Commoncrawl augmentation. Filtering removes noisy or mismatched data, while Russian Commoncrawl addresses the smaller Russian Newscrawl supply.
- Data sources: The En↔De systems use all available bitext, English and German Newscrawl, and sentence-level decoding without document boundaries.Although the language models were trained on document-level data, final decoding remained purely sentence level.
- Data sources: The En↔Ru systems use all available bitext, English and Russian Newscrawl, and filtered Russian Commoncrawl.Russian Commoncrawl augments the smaller Russian monolingual resource.
- Subword segmentation: Separate BPE encodings with 24K split operations outperform joint BPE for En↔Ru, while En↔De uses joint 32K BPE.The BPE configuration differs by language pair.
- Filtering: Language-identification filtering keeps sentence pairs with correct languages and also removes many garbage-token sentences.The procedure is applied to bitext and monolingual Newscrawl data.
- Monolingual augmentation: Moore–Lewis selection extracts a limited amount of high-quality, in-domain Russian Commoncrawl data from the larger noisy corpus.The selection targets useful monolingual data for training.
3 System Overview
The system combines large FAIRSEQ Transformers with sampled back-translation, data selection, domain-specific fine-tuning, and noisy-channel reranking. It also uses larger model capacity and language models to improve translation quality across the submission directions.
- Model architecture: The base system uses larger FAIRSEQ Transformer models, with an FFN size of 8192 adopted for subsequent models and ensembles.The authors increased embedding dimension, FFN size, number of heads, and layers, finding the larger FFN configuration a reasonable improvement while maintaining manageable network size.
- Large-scale back-translation: Back-translation augments human-translated bitext with synthetic parallel data generated from sampled translations by an ensemble of three reverse-direction models.The authors report that ensemble-generated back-translations outperform those from a single model and select a 1:1 synthetic-to-bitext training ratio.
- Commoncrawl filtering: Russian Commoncrawl data is filtered against Newscrawl using Moore and Lewis cross-entropy scoring, retaining about 5% of Commoncrawl for back-translation.Language models trained on Newscrawl and Commoncrawl score sentences by HI(s)−HN(s), with a cutoff of 0.01.
- Fine-tuning: Domain-specific fine-tuning adds one training epoch on smaller in-domain corpora selected separately for German-English, English-German, and English-Russian directions.The fine-tuning data includes prior-year test sets and News-Commentary, while other test sets remain held out for tuning procedures and evaluation metrics.
- Noisy-channel reranking: Noisy-channel reranking selects among beam-generated n-best hypotheses using forward, channel, and target-language model scores with tuned weights and a length penalty.The forward models are ensembles of fine-tuned and back-translated models; reverse-direction forward models provide channel models, and target-language models are large Transformer decoders.
4 Results
Across all four directions, progressively stronger filtering, back-translation, fine-tuning, ensembling, and reranking improve translation quality, with reranking consistently contributing substantial gains. Human evaluation ranks Facebook-FAIR first in every direction, while En→De significantly surpasses other systems and human translations.
- English→German: 1.5 BLEU: langid filtering, larger FFN, and ensembling improve En→De baseline performance on news2018.
- English→German: 3 BLEU: combining back-translated data with fine-tuning and ensembling improves En→De single-model performance.Back-translated data alone adds 0.3 BLEU, while reranking adds another 1.4 BLEU.
- German→English: Over 2.5 BLEU: back-translated data improves German→English single-model performance, with reranking contributing 1.5 BLEU of later gains.Fine-tuning, ensembling, and reranking add an additional 2.4 BLEU.
- English→Russian: 3.5 BLEU: back-translated data improves English→Russian over a bitext-only model, while Commoncrawl adds 0.2 BLEU.Fine-tuning, ensembling, and reranking add 2.2 BLEU, including 1 BLEU from reranking.
- Russian→English: 3 BLEU: back-translation improves Russian→English, while fine-tuning, ensembling, and reranking add almost 4 BLEU.The initial filtering and ensembling stage improves the bitext-only system by 1.6 BLEU, and reranking contributes 1.2 BLEU.
- Cross-direction findings: Reranking significantly improves every language direction, with gains ranging from 1 BLEU in En→Ru to 1.5 BLEU in De→En.The authors associate the smaller En→Ru gain with a relatively weak Russian language model trained on less data.
- Human evaluations: Facebook-FAIR ranks first in all four human-evaluation directions, and En→De significantly outperforms other systems and human translations.The evaluation uses different direct-assessment configurations, including source-based and target-based assessment, with some systems evaluated using document context.
- Human evaluations: Sentence-level systems perform well with or without document context; En→De ranks first in document-level rankings, while En→Ru ties human translations for first.De→En achieves the second-highest score among constrained systems.
5 Conclusions
The submission applies a common pipeline of bitext filtering, sampled back-translation, model training, fine-tuning, ensembling, and noisy-channel reranking across four directions. This approach demonstrates effective reranking on strong systems and yields first-place human-evaluation rankings in every direction.
- The same strategy filters bitext, samples back-translations, and trains strong models on combined data for all four translation directions.
- Fine-tuning and ensembling produce the final system, which decodes with noisy-channel model reranking.
- Noisy-channel reranking remains effective on top of very strong systems, and the submissions rank first in all four human-evaluation directions.