Source-linked AI summary
On Synthetic Data for Back Translation
Jiahao Xu, Yubin Ruan, Wei Bi, Guoping Huang, Shuming Shi, Lihui Chen, Lemao Liu
TL;DR
Prior back-translation work commonly uses beam search or random sampling to generate synthetic data, but the role of that data in performance has been little studied. The paper identifies quality and importance as relevant factors and proposes a method to balance them. It reports that the resulting system outperforms standard beam- and sampling-based baselines on three WMT14 tasks.
Problem
Prior work commonly generates synthetic data with beam search or random sampling, while the consequences of synthetic-corpus choice for final back-translation performance have been little studied.
Source: §1 · p. 1Method
The paper theoretically and empirically analyzes synthetic-data quality and importance, then proposes a generation method intended to balance both factors.
Source: §1 · p. 2Results
The proposed method consistently outperforms standard sampling- and beam-search-based baselines by a significant margin on three WMT14 tasks.
Source: §1 · p. 2Takeaways & Limitations
The paper concludes that quality and importance both matter for back translation and that balancing them can boost back-translation performance.
Source: §7 · p. 9The experiments follow a standard configuration with one synthetic source per target sentence and a single pass through both steps.
Source: §2 · p. 2
Abstract
from arXiv · showhide
Back translation (BT) is one of the most significant technologies in NMT research fields. Existing attempts on BT share a common characteristic: they employ either beam search or random sampling to generate synthetic data with a backward model but seldom work studies the role of synthetic data in the performance of BT. This motivates us to ask a fundamental question: {\em what kind of synthetic data contributes to BT performance?} Through both theoretical and empirical studies, we identify two key factors on synthetic data controlling the back-translation NMT performance, which are quality and importance. Furthermore, based on our findings, we propose a simple yet effective method to generate synthetic data to better trade off both factors so as to yield a better performance for BT. We run extensive experiments on WMT14 DE-EN, EN-DE, and RU-EN benchmark tasks. By employing our proposed method to generate synthetic data, our BT model significantly outperforms the standard BT baselines (i.e., beam and sampling based methods for data generation), which proves the effectiveness of our proposed methods.
1 Introduction
Prior back-translation work commonly generates synthetic data with beam search or random sampling, while the effects of that data on final performance have been little studied. The paper examines synthetic-data quality and importance and proposes a method to balance them, reporting gains over standard baselines on three benchmark tasks.
1 Introduction
- Prior work commonly uses beam search or random sampling, while few studies examine how synthetic corpora influence final back-translation performance.Source: §1 · p. 1
- Conflicting observations about high- and low-quality synthetic corpora motivate asking what kind of synthetic data contributes to back-translation performance.Source: §1 · p. 1
- The proposed method significantly outperforms beam- and sampling-based synthetic-data baselines on three WMT14 tasks.The authors report that this result supports the effectiveness of their approach.Source: §1 · p. 2
- Quality and importance weight are identified as two key factors affecting NMT performance.The authors find that the factors can be mutually exclusive to some extent, making judgments based on either one alone potentially misleading.Source: §1 · p. 2
- The proposed synthetic-corpus generation method aims to better balance quality and importance.Source: §1 · p. 2
2 Revisiting Back Translation
Back translation uses target-language monolingual data to create synthetic bilingual examples, then combines them with authentic data to train the forward model. The paper follows a standard single-pass setup and revisits the common use of beam search or random sampling for synthetic-data generation.
2 Revisiting Back Translation
- Back translation first generates synthetic bilingual data by back-translating target-language sentences, then trains the forward model on authentic and synthetic corpora.Source: §2 · p. 2
- The standard configuration generates one synthetic source per target sentence and performs both steps in a single pass.The authors follow this setup for generality and say their idea can also apply to other configurations.Source: §2 · p. 2
- Common synthetic-data strategies are deterministic beam search and random sampling with the backward model.Source: §2 · p. 2
- Contradictory findings about synthetic-corpus quality motivate investigating what kind of corpus benefits back translation.Source: §2 · pp. 2–3
3 Understanding Synthetic Data by Two Factors
The paper analyzes back translation through the target-language marginal-likelihood objective and identifies synthetic-data quality and importance as relevant factors. Empirical comparisons show that quality alone does not consistently track back-translation performance.
3.1 Theoretical Explanation
- The marginal-likelihood derivation links the back-translation objective to a multiplicative importance weight alongside the forward-model likelihood term.The importance weight is constant with respect to the forward-model parameter θ.Source: §3.1 · p. 3
- The analysis identifies translation quality and importance weight as two factors relevant to maximizing the marginal-likelihood objective.Source: §3.1 · p. 3
- The authors explain that higher-quality candidates can have lower importance weights, making the factors mutually exclusive to some extent.They use this trade-off to explain why high-quality synthetic data may sometimes correspond to better testing performance and sometimes worse performance.Source: §3.1 · p. 3
- The paper estimates synthetic-data quality with BLEU when references are available, or backward-model log likelihood as a surrogate otherwise.It estimates importance using a source-language model trained on a large monolingual corpus.Source: §3.1 · p. 3
3.2 Empirical Justification
- On WMT14 DE-EN, beam has higher synthetic-corpus quality than sampling but worse test performance; comparisons across EN-DE and RU-EN also show quality alone is insufficient.The paper additionally reports that beam performs better than beam* when their importance weights are almost similar and beam quality is better.Source: §3.2 · p. 4, Table 1 · p. 4, Table 2 · p. 4
4 Improving Synthetic Data for BT
Because synthetic-data quality and importance can conflict, the paper proposes two ways to balance them when generating back-translation data: combining beam and sampling outputs, or selecting candidates with a balancing score.
4 Improving Synthetic Data for BT
- The section proposes data manipulation and gamma score to trade off synthetic-data quality and importance, which can be mutually exclusive.Both factors are described as beneficial to back-translation performance.Source: §4 · p. 4
4.1 Data Manipulation
- The data-manipulation method splits the monolingual corpus and translates one part with beam search and the other with sampling, tuning γ to set their proportions.The combined corpus is used to train the back-translation model.Source: §4.1 · pp. 4–5
- The mixed corpus is generally lower in quality than beam-search data and lower in importance weight than sampling data.Each synthetic translation comes from either beam search or sampling.Source: §4.1 · p. 5
4.2 Gamma Score
- Gamma score selects translations by balancing quality and importance, either by choosing the highest-scoring candidate or sampling according to scores.The method samples a candidate set from the backward model and normalizes candidate scores before selection.Source: §4.2 · p. 5
- The paper reports that fixed-γ interpolation over the translation space performs poorly because the estimates are uncalibrated and the factors vary in scale across inputs.A fixed γ is therefore difficult to balance for different y.Source: §4.2 · p. 5
- Using N-best decoding to generate candidates is mentioned as an alternative that remains future work.The described method instead samples candidates from the backward model.Source: §4.2 · p. 5
5 Experiments
Experiments compare data manipulation and gamma-based synthetic-data methods with standard back-translation baselines on WMT14 DE-EN, EN-DE, and RU-EN, then analyze the generated corpora.
5.1 Settings
- Experiments use WMT14 DE-EN, EN-DE, and RU-EN, Newscrawl monolingual data, a six-block Big Transformer, and fixed training settings across models.The reported monolingual data comprise 4.5 million sentences for DE-EN and 2.5 million for RU-EN; baseline and BT models train for 400K and 1.6M steps, respectively.Source: §5.1 · pp. 5–6
5.2.1 Results on DE-EN
- With synthetic data alone, manipulation reaches 31.3 BLEU versus 29.2 for sampling and 27.6 for beam, but with bitext it only nearly matches sampling BT.The synthetic-only result is 0.7 BLEU below the bitext baseline.Source: §5.2.1 · p. 6, Table 3 · p. 6
- On DE-EN, gamma sampling beats sampling and beam BT by 0.9 and 2.3 BLEU; gamma methods gain almost 1 and 0.4 BLEU on EN-DE and RU-EN.The DE-EN comparison is reported against sampling and beam, respectively; the other-task gains are reported for EN-DE and RU-EN, respectively.Source: Table 4 · p. 6, §5.2.1 · p. 7, §5.3 · p. 7
5.4 Analysis on Synthetic Corpus
- Spectrum analysis places gamma methods between sampling and beam in representation variety, with gamma sampling richer than gamma selection; gamma sampling also produces the longest sentences.Sampling has the most uniform spectrum, beam the least variety, and gamma selection and sampling have almost equal sentence lengths.Source: §5.4 · p. 7, Figure 1 · p. 8
- Beam favors high-frequency tokens while sampling favors low-frequency tokens; gamma sampling has the same vocabulary size as sampling, possibly because it generates candidates by random sampling.Source: §5.4 · pp. 7–8
6 Related Work
Related work spans back-translation methods, data augmentation, and semi-supervised machine translation. Prior studies explore synthetic-data generation and varied uses of monolingual data.
Back-translation NMT
- Back-translation research progressed from domain adaptation in phrase-based systems to using back-translation to train NMT models, with later work exploring iterative, tagged, confidence-weighted, and diversified approaches.Studies also compared beam and sampling-based synthetic-corpus generation.Source: §6 · p. 8
Data Augmentation for NMT
- NMT data augmentation includes back-translation and approaches such as adding sentences with low-frequency words and SwitchOut, with reported improvements varying across translation tasks.Other studies examine training–test variance and input sensitivity or prediction margin.Source: §6 · p. 8
Semi-supervised Machine Translation
- Semi-supervised machine translation research leverages monolingual or unlabeled data through iterative learning, target-language model states, and studies of source- and target-side monolingual corpora.Work also reports translation-quality improvements for naturally occurring text and translationese according to professional human translators.Source: §6 · pp. 8–9
7 Conclusion
The paper studies how synthetic-data quality and importance weight relate to back-translation performance, then proposes a generation method that balances them. It identifies applying the method to other back-translation methods and broader NLP tasks as future work.
7 Conclusion
- The paper identifies synthetic-data quality and importance weight as factors in back-translation and proposes a generation method that balances both.The authors suggest applying it to other back-translation methods and tasks such as paraphrasing and style transfer.Source: §7 · p. 9
A Model Details
The appendix lists the NMT optimization settings and training command, and notes that the GPT model uses only the bitext source side without extra datasets.
A Model Details
- The NMT models use Adam with β1 = 0.9 and β2 = 0.98, a maximum learning rate of 7 × 10−4, and 4000 warmup updates.The appendix also gives the fairseq training command and model hyperparameters.Source: A Model Details · p. 11
- The NMT training command specifies the Transformer WMT English–German big architecture, 4096 maximum tokens, and 1,600,000 maximum updates.It also specifies label smoothing of 0.1 and the inverse-square-root learning-rate schedule.Source: A Model Details · pp. 11–12
- The GPT model is trained only on the source side of the bitext corpus, without extra datasets.The appendix lists its training command and core settings.Source: A Model Details · p. 12