Source-linked AI summary
Generalized Data Augmentation for Low-Resource Translation
Mengzhou Xia, Xiang Kong, Antonios Anastasopoulos, Graham Neubig
TL;DR
Low-resource translation is constrained by scarce parallel data, and sharing with a related high-resource language can still leave vocabulary or syntactic mismatches. The paper augments training data through English back-translation and a two-step pivot that converts HRL data into pseudo-LRL data. Across datasets, the methods improve over supervised back-translation.
Problem
Low-resource MT lacks the large parallel corpora needed for adequate performance, and related-language sharing can leave vocabulary or syntactic mismatches.
Source: §1 · p. 1, §2.3 · p. 3Method
The framework augments LRL translation with English back-translation and a two-step pivot that substitutes LRL words into HRL sentences before unsupervised editing.
Source: §1 · pp. 1–2Results
1.5 to 8 BLEU points over standard supervised back-translation across all datasets; combining English monolingual and HRL–English parallel data adds up to 1.1 BLEU points.
Source: §1 · p. 2Takeaways & Limitations
The authors conclude that the framework makes best use of available resources for low-resource translation.
Source: §7 · p. 9The pivot rationale assumes the languages are related enough for a high-quality bilingual dictionary and that relatively high-resource HRL–English data exists.
Source: §3.3 · p. 5
Abstract
from arXiv · showhide
Translation to or from low-resource languages LRLs poses challenges for machine translation in terms of both adequacy and fluency. Data augmentation utilizing large amounts of monolingual data is regarded as an effective way to alleviate these problems. In this paper, we propose a general framework for data augmentation in low-resource machine translation that not only uses target-side monolingual data, but also pivots through a related high-resource language HRL. Specifically, we experiment with a two-step pivoting method to convert high-resource data to the LRL, making use of available resources to better approximate the true data distribution of the LRL. First, we inject LRL words into HRL sentences through an induced bilingual dictionary. Second, we further edit these modified sentences using a modified unsupervised machine translation framework. Extensive experiments on four low-resource datasets show that under extreme low-resource settings, our data augmentation techniques improve translation quality by up to~1.5 to~8 BLEU points compared to supervised back-translation baselines
1 Introduction
The paper addresses low-resource MT through augmentation that uses resources from a related high-resource language (HRL) alongside English monolingual data. It proposes pivoting HRL–English data into pseudo-LRL–English data and reports gains over supervised back-translation.
1 Introduction
- Low-resource MT lacks the large parallel corpora needed for adequate performance, while related-language sharing can leave vocabulary and syntactic mismatches and add model complexity.The introduction cites transfer from HRL to LRL and joint HRL–LRL training as established approaches.Source: §1 · p. 1
- The framework augments low-resource translation through target-language back-translation and by converting HRL–English data into pseudo-LRL–English data.For pivoting, it substitutes LRL words into HRL sentences using an induced bilingual dictionary, then edits them with unsupervised machine translation.Source: §1 · pp. 1–2
- The evaluated methods improve over simple unsupervised HRL-to-LRL translation by more than 2 to 10 BLEU points across four language pairs.The paper describes a thorough empirical evaluation using accessible data.Source: §1 · p. 2
- 1.5 to 8 BLEU points over standard supervised back-translation across all datasets; using both English monolingual data and HRL–English parallel data adds up to 1.1 BLEU points.The reported comparison spans all datasets.Source: §1 · p. 2
2 A Generalized Framework for Data Augmentation
The framework combines back-translation from English with pivoting through a related HRL to create pseudo-LRL training data. Its setup draws on parallel and monolingual resources across the LRL, HRL, and English.
2.1 Datasets and Notations
- The framework assumes access to LRL–English, HRL–English, and LRL–HRL parallel data, plus monolingual data in the LRL, HRL, and English.The notation uses S and T for the source and target sides of parallel data and M for monolingual data.Source: §2.1 · p. 2
2.2 Augmentation from English
- English monolingual data can be back-translated into either the LRL or HRL; HRL back-translation is proposed as potentially more effective because its generated data may be higher quality and the languages are related.The authors note that limited training data can make generated LRL data low quality and harm LRL–English performance.Source: §2.2 · pp. 2–3
2.3 Augmentation via Pivoting
- Pivoting converts the source side of HRL–English data into LRL to create pseudo-LRL–English data, addressing vocabulary and possible syntax mismatch between related languages.The paper presents conversion using HRL-to-LRL translation and characterizes that translation as an easier task than translating from English.Source: §2.3 · p. 3
- A second pivoting route first back-translates English monolingual data into HRL, then converts those HRL sentences into LRL.This route creates a pseudo-LRL–English dataset from English monolingual data.Source: §2.3 · p. 3
- The paper proposes more reliable HRL-to-LRL conversion methods because supervised conversion from limited LRL–HRL data can still produce poor augmented data.The motivation draws on morphological and word-order similarities between an LRL and its corresponding HRL.Source: §2.3 · p. 3
3 LRL-HRL Translation Methods
The section presents two ways to convert related-language HRL data into pseudo-LRL data: dictionary-based word substitution and modified unsupervised machine translation. It also motivates pivoting through an HRL as a way to split a difficult direct translation into two easier steps.
3.1 Augmentation with Word Substitution
- An induced bilingual dictionary maps related-language word embeddings, enabling HRL words to be replaced with LRL words and produce pseudo-LRL text.The mapping is learned as a Procrustes problem; words absent from the dictionary remain unchanged during corpus construction.Source: §3.1 · p. 3
3.2 Augmentation with Unsupervised MT
- Word substitution alone may not reproduce actual LRL data, and unsupervised translation between low-resource, morphologically rich languages remains challenging.The section notes that prior UMT results were exceptionally poor for dissimilar language pairs and that LRL monolingual corpora are smaller than HRL corpora.Source: §3.2 · p. 4
- Modified UMT further translates word-substituted pseudo-LRL data, using denoising auto-encoding and iterative back-translation without synthetic parallel initialization.Its initialization substitutes HRL words, learns joint segmentation for HRL and pseudo-LRL monolingual data, then trains UMT between them; injected LRL words increase vocabulary overlap with true LRL.Source: §3.2 · p. 4
3.3 Why Pivot for Back-Translation?
- Pivoting can be preferable to direct English-to-LRL back-translation when HRL and LRL support dictionary induction, HRL-English data are relatively plentiful, and an LRL-English dictionary is hard to obtain.The proposed path separates English-to-HRL translation, supported by available data, from HRL-to-LRL translation, supported by language relatedness.Source: §3.3 · pp. 4–5
4 Experimental Setup
The experiments use four LRL–HRL pairs and parallel and monolingual resources, then train supervised and unsupervised Transformer models. Model selection relies on development performance, with mixed fine-tuning after base training.
4.1 Data
- Experiments use TED data for Azerbaijani, Belarusian, Galician, and Slovak alongside related Turkish, Russian, Portuguese, and Czech, with small LRL-HRL parallel sets and monolingual Wikipedia data.For Azerbaijani, Belarusian, and Galician, all available Wikipedia data are used; other languages use similarly sized samples.Source: §4.1 · p. 5
4.2 Pre-processing
- Each LRL-HRL pair uses a joint SentencePiece model trained on concatenated monolingual corpora, while English segmentation is trained on English data alone.The vocabulary size is 20K for each model.Source: §4.2 · p. 5
4.3 Model Architecture
- Supervised and unsupervised systems use Transformer architectures with matching layer sizes and dimensions.The supervised model has four encoder and decoder layers with 256-dimensional embeddings and hidden units; the unsupervised model shares the first three encoder and decoder layers.Source: §4.3 · p. 5
4.4 Training and Model Selection
- Training first uses concatenated LRL-English and HRL-English data, then mixed fine-tuning on base and augmented data; development performance selects the best models.Each setting is trained until development perplexity converges.Source: §4.4 · pp. 5–6
5 Results and Analysis
The augmentation methods improve low-resource translation across the tested settings, with the strongest results from combining complementary sources of HRL–ENG data. The analysis links rare-word coverage to translation quality and finds that modified unsupervised MT improves pivot translation over standard unsupervised MT.
5.1 Baselines
- The base supervised model ranges from 11.8 to 29.5 BLEU, while standard unsupervised MT scores extremely low and supervised back-translation yields at best modest improvements.SLK–ENG is a notable back-translation exception, with more parallel training data than the other settings.Source: §5.1 · p. 6
- Supervised back-translation into the HRL helps more than into the LRL across all four languages, while back-translation into the LRL can worsen BEL and GLG results.The authors attribute the weak LRL back-translation results to insufficient data for training a good LRL–ENG model.Source: §5.1 · p. 6
5.2 Back-translation from HRL
- The proposed augmentation strategies significantly outperform prior modeling and multilingual-target methods, by +3.0 BLEU in AZE, +7.8 in BEL, +2.3 in GLG, and +1.1 in SLK.Combining word-substitution and modified-UMT datasets further improves results in all four cases.Source: §5.2 · p. 7
- Two-step pivot back-translation from English through the HRL still beats standard back-translation, although it performs slightly worse than augmentation from parallel HRL–LRL data.For the second pivot step on back-translated HRL, word substitution is preferable to modified UMT.Source: §5.2 · p. 7, §5.3 · p. 8
- Combining word substitution on parallel and back-translated HRL–ENG data yields 3.6 to 8.2 BLEU points over the base model and 0.3 to 2.1 over the best HRL–ENG results.Using two distinct English data resources also aids the decoder’s language modeling capabilities.Source: §5.2 · p. 7
5.3 Analysis
- Modified UMT scores 2 to 10 pivot-BLEU points above standard UMT, but pivot BLEU and resulting translation BLEU are imperfectly correlated.Combining modified-UMT and word-substitution datasets consistently improves on word substitution alone, suggesting complementary effects.Source: §5.3 · pp. 7–8
- Rare-word address rate and LRL–ENG BLEU are highly correlated across augmentation methods, indicating that the methods mitigate rare-word problems.A test-set word is rare when it falls in the training set’s lowest 10th frequency percentile; it is addressed when augmentation moves it above that threshold.Source: §5.3 · p. 8
- Bidirectional dictionary induction performs better than unidirectional induction in most cases, while its injected word-count ratio exceeds its word-type ratio, indicating relatively frequent injected words.The exception to the usual BLEU advantage is BEL; adding each word’s closest neighbor may introduce harmful noise.Source: §5.3 · pp. 8–9
6 Related Work
The related work situates the paper among multilingual and unsupervised translation, bilingual dictionary induction, and pivoting approaches. It describes prior methods across these areas and distinguishes the paper’s data-augmentation setting.
6 Related Work
- Multilingual low-resource MT shares parameters across languages through mapped word embeddings, character n-gram embeddings, or missing-data augmentation.Source: §6 · p. 9
- Unsupervised MT trains NMT without parallel data (Artetxe et al., 2018; Lample et al., 2018a,c), with later work improving the framework and applying it to tasks beyond MT.Ren et al. (2019) used statistical MT as posterior regularization for en-fr and en-de MT.Source: §6 · p. 9
- Prior bilingual-dictionary work includes low-resource named-entity recognition and information retrieval; Hassan et al. (2017) synthesized dialect-translation data using word embeddings.That synthesis required both LRL–ENG and HRL–LRL dictionaries, whereas this paper uses only an HRL–LRL dictionary.Source: §6 · p. 9
- Pivoting from phrase-based MT was later adapted to neural MT, including joint training, guidance from pivot-target models, and iterative zero-shot translation.The cited approaches include De Gispert and Marino (2006), Cohn and Lapata (2007), Levinboim and Chiang (2015), Cheng et al. (2017), Chen et al. (2017), and Lakew et al. (2018).Source: §6 · p. 9
7 Conclusion
The paper proposes generalized data augmentation for low-resource translation, including two-step pivoting to convert HRL parallel data to LRL. Future work will address dictionary quality and jointly train the pivoting and low-resource translation systems.
7 Conclusion
- The conclusion proposes generalized low-resource translation augmentation and a two-step pivoting method to convert HRL parallel data to LRL.Source: §7 · p. 9
- Future work will explore controlling induced-dictionary quality to improve word substitution and modified UMT.Source: §7 · p. 9
- The authors also plan an end-to-end iterative framework jointly training the M-UMT pivoting and low-resource translation systems to leverage more augmented-data versions.Source: §7 · p. 9