Source-linked AI summary
Investigating Backtranslation in Neural Machine Translation
Alberto Poncelas, Dimitar Shterionov, Andy Way, Gideon Maillette de Buy Wenniger, Peyman Passban
TL;DR
Limited high-quality parallel data has made backtranslation common, but its effects on NMT performance remain uncertain. This paper compares authentic-only, synthetic-only, and hybrid German-to-English systems as synthetic data increases, finding gains that eventually level off and can decline when synthetic data dominates.
Problem
The study addresses uncertainty about how back-translated data affects NMT performance when used alone or with human-generated parallel data.
Method
The authors train German-to-English NMT systems on authentic-only, synthetic-only, and hybrid corpora while systematically increasing the amount and ratio of back-translated data.
Results
A 1.2% performance drop occurred for the 3.5M hybrid model compared with the 3M hybrid model, while synthetic-only systems performed close to authentic-only systems.
Takeaways & Limitations
Backtranslation can improve NMT quality, including in resource-poor scenarios, but performance may level off when synthetic data becomes too dominant.
Takeaways & Limitations
The findings are based on German-to-English experiments, and the authors plan to test other language pairs and domains.
Abstract
from arXiv · showhide
A prerequisite for training corpus-based machine translation (MT) systems -- either Statistical MT (SMT) or Neural MT (NMT) -- is the availability of high-quality parallel data. This is arguably more important today than ever before, as NMT has been shown in many studies to outperform SMT, but mostly when large parallel corpora are available; in cases where data is limited, SMT can still outperform NMT. Recently researchers have shown that back-translating monolingual data can be used to create synthetic parallel corpora, which in turn can be used in combination with authentic parallel data to train a high-quality NMT system. Given that large collections of new parallel text become available only quite rarely, backtranslation has become the norm when building state-of-the-art NMT systems, especially in resource-poor scenarios. However, we assert that there are many unknown factors regarding the actual effects of back-translated data on the translation capabilities of an NMT model. Accordingly, in this work we investigate how using back-translated data as a training corpus -- both as a separate standalone dataset as well as combined with human-generated parallel data -- affects the performance of an NMT model. We use incrementally larger amounts of back-translated data to train a range of NMT systems for German-to-English, and analyse the resulting translation performance.
1 Introduction
NMT quality depends strongly on training-data quality and quantity, while limited parallel data motivates backtranslation. This paper systematically tests authentic, synthetic, and hybrid data, including increasing synthetic-to-authentic ratios.
- NMT generally requires more training data than SMT to achieve good translation performance.
- Backtranslating freely available monolingual data has become a standard way to create synthetic parallel data when good-quality parallel data is insufficient.
- The paper questions whether backtranslation became standard without adequate consideration of its merits, given concerns about machine-translated errors and data quality.
- The study compares NMT systems trained on authentic-only, synthetic-only, and hybrid corpora of systematically increasing sizes.
- For hybrid systems, the authors increase the back-translated-to-human data ratio to test whether synthetic data eventually harms translation performance.
2 Related Work
Prior work explores monolingual-data integration, synthetic-only training, and backtranslation across MT settings. These approaches differ in whether they modify model architecture, use synthetic data alone, or apply backtranslation within statistical or neural systems.
- Several studies improve NMT with monolingual data by integrating an external language model or modifying the decoder architecture.
- Backtranslation became a popular NMT training step after studies reported good results from adding synthetic data.
- Park et al. analyze synthetic-only NMT using source-side synthetic data, target-side synthetic data, or mixtures containing synthetic sentences.
- Backtranslation has also improved overall translation quality in PBSMT across 8 language pairs.
3 Issues involved in creating back-translated parallel data
The section examines whether errors in synthetic data limit NMT quality as synthetic data becomes dominant. It motivates a systematic baseline analysis of authentic-only, synthetic-only, and mixed training data.
- Machine-generated translation errors may be learned and replicated by models trained on synthetic data.
- The authors hypothesize that NMT performance will degrade when synthetic data becomes overly dominant in training.
- The study varies parallel-dataset size and the amount of back-translated data to analyze progressive effects on NMT systems.
- The proposed experimental baseline controls a freely available-data setup while acknowledging other factors that may affect results.
- The analysis compares authentic-only and synthetic-only systems, testing the hypothesis that synthetic data alone cannot produce good performance.
4 Data
The experiments use WMT 2015 German–English data and compare authentic-only, synthetic-only, and hybrid training configurations. Hybrid systems retain 1M authentic sentence pairs while varying the amount of back-translated data.
- The German–English WMT 2015 corpus contains 4.48M cleaned sentence pairs, totaling 225M words.
- The study evaluates authentic-only models as a baseline for translation quality.
- Synthetic-only models use back-translated data when parallel data is unavailable and may also support domain adaptation.
- Hybrid models combine 1M authentic sentence pairs with differing amounts of back-translated data to trace quality changes as the synthetic-to-authentic ratio increases.
- All models are evaluated on the same WMT 2015 news test set containing 2169 tokenized and truecased sentences.
5 Experimental set-up
The experiments use a fixed, representative OpenNMT configuration to compare authentic, synthetic, and hybrid German-to-English training data under controlled corpus constructions. The setup prioritizes isolating backtranslation effects and enabling replication over maximizing absolute system performance.
- Model configuration: The study uses recurrent neural-network sequence-to-sequence NMT models with attention, implemented in OpenNMT’s PyTorch port.
- Model configuration: The researchers hold the NMT setup static to simplify direct comparisons of backtranslated-data effects and support replication.They explicitly do not aim to build the absolutely best possible systems.
- Scope: The experiments were not intended to exhaust the many possible NMT parameter settings or architectures, and additional experiments were outside the paper’s scope.
- Model configuration: The model uses a 2-layer LSTM with 500 hidden units, source and target vocabularies of 50,002 and 50,004, 13 training epochs, stochastic gradient descent, and batch size 64.
- Data construction: A 1,000,000-sentence authentic German-to-English baseline and a reversed-direction English-to-German model provide the basis for generating synthetic data.Preliminary human and automatic evaluations found the 1M-sentence model sufficient for ‘good enough’ translations.
- Data construction: The auth0+ corpora grow from 1,000,000 to 3,500,000 authentic sentences, while hybr combines the first 1,000,000 authentic sentences with successive 500,000-sentence backtranslated subsets.
- Data construction: The auth1+ corpora begin at sentence 1,000,000 and grow by 500,000 sentences, with corresponding synth sets consisting of their backtranslated data.
- Data construction: The comparisons preserve the same original data while varying whether it is authentic, partially backtranslated, or fully backtranslated.
6 Results
Increasing authentic, synthetic, and hybrid training data affects NMT quality differently: synthetic data helps initially but plateaus, while hybrid data improves quality before degrading when synthetic data dominates.
- Authentic Data Models: BLEU rose from 0.2278 with 1M authentic sentence pairs to 0.2446 with 3M, then fell slightly at 3.5M.The absolute improvement from 1M to 3M was 0.0168, or 7.4% relative; other evaluation metrics corroborated the pattern.
- Synthetic Data Models: Synthetic-only BLEU increased from 0.229 with 1M sentence pairs to 0.2363 with 3.5M, a 0.0073 absolute or 3.2% relative improvement.TER, METEOR, and CHRF showed a more mixed pattern than BLEU.
- Synthetic Data Models: Synthetic-only systems initially approached authentic-only quality, but their quality increased less as more back-translated sentences were added.The gap between synthetic and authentic systems widened at larger data sizes, indicating limited scalability.
- Synthetic Data Models: Synthetic-model coverage stayed near 61% across training sizes, below authentic-model coverage above 66%.Adding more synthetic data did not increase coverage, whereas authentic models exceeded 66% at all training sizes.
- Synthetic Data Models: The back-translation system limits synthetic-only performance through restricted source vocabulary and errors, causing the learned knowledge to reach a steady point.The synthetic-only NMT system can nevertheless learn the original system’s encoded knowledge and even exceed its quality.
- Hybrid Data Models: Adding 0.5M synthetic sentences to 1M authentic sentences outperformed both the 1M authentic-only model and the same-sized authentic-only comparison at 1.5M sentences.The reported result supports benefits from combining authentic and back-translated data.
- Hybrid Data Models: The best hybrid combination used 1M authentic and 2M synthetic sentence pairs, reaching BLEU 0.2442 and improving 2.8% relative to the smallest hybrid model.The absolute improvement was 0.0066 BLEU points.
- Hybrid Data Models: Hybrid coverage exceeded synthetic-only coverage but declined as synthetic data grew, while quality gains became less steep at 2M total sentences or more.The authors attribute declining coverage to synthetic vocabulary increasingly displacing words frequent in authentic parallel data.
7 Conclusion and Future Work
The study finds that increasing synthetic data generally improves German-to-English NMT performance, but gains can tail off when synthetic data dominates. Synthetic-only training performs surprisingly well, while broader validation remains future work.
- Translation performance generally improves as more synthetic data is added, but appears to plateau when synthetic data becomes dominant.The 3.5M hybrid model performs 1.2% worse than the 3M hybrid model.
- The experiments suggest a possible tipping point in the balance between synthetic and authentic training data.Future work will test whether an optimal synthetic-to-authentic ratio exists beyond which additional synthetic data reduces quality.
- Synthetic-only training performs surprisingly well, with quality close to authentic-only engines.With 1M sentences, the synthetic-only engine even outperforms the same-sized authentic-only engine on METEOR.
- The findings have particular implications for resource-poor scenarios where a reverse-direction engine can generate synthetic data for bootstrapping.Such an engine can create arbitrarily large amounts of back-translated data for the other language direction.
- Future experiments will examine artificial data across additional language pairs and domains.