Source-linked AI summary
Understanding Back-Translation at Scale
Sergey Edunov, Myle Ott, Michael Auli, David Grangier
TL;DR
Neural machine translation needs parallel data, but monolingual target text is far more abundant. This paper studies scalable back-translation and alternative synthetic-source generation methods, finding that sampling or noised beam outputs are generally strongest and achieving 35 BLEU on WMT’14 English-German.
Problem
Parallel corpora are limited, motivating methods that use the much larger supply of target-language monolingual data for neural machine translation.
Method
The paper augments genuine bitext with target sentences back-translated into synthetic sources generated by sampling, beam search, greedy search, or noised beam outputs.
Results
Sampling or noised beam outputs are most effective in all but resource poor settings, and the approach reaches 35 BLEU on WMT’14 English-German using public benchmark data.
Takeaways & Limitations
Sampling and noisy synthetic data provide a richer training signal than beam or greedy outputs, while synthetic data can reach up to 83% of real-bitext performance.
Takeaways & Limitations
The paper leaves end-to-end optimization of the back-translation model for future work.
Abstract
from arXiv · showhide
An effective method to improve neural machine translation with monolingual data is to augment the parallel training corpus with back-translations of target language sentences. This work broadens the understanding of back-translation and investigates a number of methods to generate synthetic source sentences. We find that in all but resource poor settings back-translations obtained via sampling or noised beam outputs are most effective. Our analysis shows that sampling or noisy synthetic data gives a much stronger training signal than data generated by beam or greedy search. We also compare how synthetic data compares to genuine bitext and study various domain effects. Finally, we scale to hundreds of millions of monolingual sentences and achieve a new state of the art of 35 BLEU on the WMT'14 English-German test set.
1 Introduction
The paper studies back-translation as a way to exploit abundant target-language monolingual data when parallel corpora are limited, scaling the method to hundreds of millions of sentences. It reports strong benchmark results using public data alone.
- Motivation: The approach addresses the mismatch between limited parallel corpora and the much larger supply of monolingual data available for machine translation.Earlier uses of monolingual data primarily improved language-model fluency.
- Method: Back-translation converts target-language monolingual sentences into synthetic source sentences, then adds them to genuine bitext for training the final source-to-target NMT system.An intermediate target-to-source model generates the synthetic sources.
- Scope: The study broadens prior back-translation analysis by investigating multiple ways to generate synthetic sources and scaling experiments to hundreds of millions of sentences.The experiments use strong baseline models trained on public WMT bitext.
2 Related work
Prior work uses monolingual data through language models, back-translation, dual learning, and related semi-supervised or multilingual approaches. Back-translation is presented as a simple way to augment bitext across translation settings.
- Neural machine translation: Neural machine translation commonly uses an encoder-decoder architecture trained to maximize target-sentence likelihood given parallel source sentences.The decoder generates target text left to right at inference.
- Monolingual data: Monolingual data has supported translation through target-side language models, model integration, multitask learning, and parameter sharing.These approaches primarily improve fluency or integrate language-model information into NMT.
- Back-translation: Back-translation leverages target-language monolingual data by generating synthetic source sentences and adding the resulting data to human bitext.It has been applied to phrase-based translation, NMT, domain adaptation, and unsupervised MT.
- Related approaches: Other studies explore copied-target synthetic sources, sampled sources, iterative refinement, source-side monolingual data, multilingual transfer, and data augmentation.Sampling has been reported as more effective than beam search in concurrent work.
- Extensions: Dual learning extends back-translation by jointly training source-to-target and target-to-source systems and applying the procedure in both directions.Related work also applies multiple rounds of back-translation in unsupervised NMT.
3 Generating synthetic sources
The paper argues that beam and greedy decoding produce overly regular synthetic sources because they concentrate on high-probability outputs. It therefore studies unrestricted and restricted sampling alongside noisy beam outputs.
- Motivation: Beam and greedy search focus on the distribution’s head, producing regular synthetic sources that may not cover the true data distribution.The paper links this limitation to MAP prediction favoring the most likely alternative under ambiguity.
- Sampling methods: The study compares unrestricted sampling, top-k restricted sampling, and noise applied to beam-search outputs as alternatives to MAP decoding.Restricted sampling renormalizes the k most likely tokens at each step before sampling.
- Noised beam: Noising transforms beam outputs by deleting words with probability 0.1, replacing words with a filler token with probability 0.1, and randomly swapping tokens.The procedure is motivated by denoising approaches used in autoencoder setups.
4 Experimental setup
The experiments use filtered WMT English-German and English-French bitext with large target-language news monolingual corpora. Models share a Big Transformer configuration and are evaluated mainly with tokenized BLEU.
- English-German data: The English-German setup contains 5.18M filtered bitext sentence pairs and 226M deduplicated German monolingual sentences.Filtering removes sentences longer than 250 words and pairs with source/target length ratios above 1.5.
- English-French data: The English-French setup contains 35.7M filtered sentence pairs and 31M monolingual newscrawl sentences after language identification.Evaluation uses newstest2012 for development and newstest2013-2015 for final results.
- Evaluation: Results are reported mainly with case-sensitive tokenized BLEU, with detokenized BLEU also computed using sacreBLEU.The paper distinguishes tokenized and detokenized reporting.
- Model configuration: All experiments use a PyTorch fairseq implementation of the Big Transformer with six encoder and six decoder blocks.The shared configuration includes 1024-dimensional representations, 4096-dimensional feed-forward layers, and 16 attention heads.
5 Results
The results compare synthetic-source generation methods for back-translation and analyze why sampling and noisy beam outputs provide stronger training signals than greedy or beam search.
- 5.1 Synthetic data generation methods: Sampling and beam+noise outperform greedy and beam search by 0.8–1.1 BLEU, while restricted top10 sampling is intermediate but less effective than unrestricted sampling or beam+noise.Across wider WMT English-German test sets, sampling and beam+noise perform roughly equally, motivating the choice of sampling for later experiments.
- 5.2 Analysis of generation methods: Sampling can produce inadequate translations, including target words without source counterparts, because it sometimes selects highly unlikely outputs.This quality trade-off contrasts with the stronger training signal associated with sampling and noisy beam outputs.
- 5.2 Analysis of generation methods: Sampling and noisy beam outputs provide richer training signals because they expose models to more varied source sentences than regular argmax outputs.The analysis links beam-search regularity to weaker training signals and suggests that variation improves robustness to natural reordering and substitutions.
- 5.2 Analysis of generation methods: Beam-generated synthetic data is easier to fit and more predictable than sampling, noisy beam, and genuine source data, indicating lower output richness.Beam-data perplexity falls below 2 after five epochs, while a language model assigns higher probability to beam outputs than to sampling, beam+noise, or real source sentences.
1. On 640K sentences pairs, we train a back-
The experiments compare synthetic-data generation, data domain, bitext mixing, and scale, finding that sampling-based back-translation is generally effective but resource-poor settings favor beam outputs.
- 5.3 Low resource vs. high resource setup: Sampling is more effective than beam search with 640K and 5.2M bitext, whereas beam is better with 80K because sampling noise harms very low-resource systems.German-English back-translation accuracy rises from 13.5 BLEU with 80K bitext to 24.3 with 640K and 28.3 with 5M.
- 5.4 Domain of synthetic data: BT-news improves newstest2012 by 2.6 BLEU and performs almost as well as real bitext, while BT-bitext achieves 83% of the real-bitext improvement.These results support synthetic data as nearly as effective as human-translated data when domains match.
- 5.4 Domain of synthetic data: On valid-mixed, BT-news improves the baseline by up to 1.2 BLEU, while BT-bitext improves it by 2.7 BLEU and trails real bitext by 1.3 BLEU.The domain-matched BT-bitext achieves 67% of the gain from real human bitext.
- 5.5 Bitext upsampling: Beam and greedy synthetic data benefit from bitext upsampling, whereas sampling and beam+noise require no upsampling because they provide a stronger training signal.The comparison uses 5M bitext sentences and 24M synthetic sentences.
- 5.6 Large-scale results: The English-French experiment improved the best reported newstest2014 baseline by 2.6 tokenized BLEU after adding 31M sampled newscrawl sentences.Training used 35.7M bitext sentence pairs and did not upsample the bitext.
- 6. Large-scale results: 35 BLEU on WMT’14 English-German was achieved using 226M monolingual sentences and back-translation with sampling.The result used only WMT benchmark data and established a new state of the art.
6 Submission to WMT’18
The WMT’18 English-German submission ensembled six back-translation models trained with large-scale monolingual data and varied bitext upsampling ratios.
- 6 Submission to WMT’18: The WMT’18 English-German entry ensembled six back-translation models trained on all available bitext plus 226M newscrawl sentences.Four models used bitext upsampling ratio 16, while the remaining models used ratios 32 and 8.
- 6 Submission to WMT’18: Source-copy outputs were detected using a unigram Jaccard similarity threshold of 0.5, identifying about 0.5% of outputs for replacement.Replacement used a model trained only on the news-commentary portion of WMT’18.
7 Conclusions and future work
Back-translation improves neural machine translation, with sampling and noised beam outputs providing stronger gains than conventional decoding and enabling a 35 BLEU state-of-the-art result. Future work will optimize the back-translation model directly for the final forward model.
- 35 BLEU establishes a new state-of-the-art result on WMT’14 English-German using publicly available benchmark data only.
- Sampling or noised beam outputs outperform pure beam search by 1.7 BLEU on average for WMT English-German translation.These methods provide a richer training signal except in resource-poor settings.
- Future work will optimize the back-translation model end to end so its synthetic sources are most helpful to the final forward model.