PaperlayerView exact revision on arXiv

Source-linked AI summary

Survey of Low-Resource Machine Translation

Barry Haddow, Rachel Bawden, Antonio Valerio Miceli Barone, Jindřich Helcl, Alexandra Birch

arXiv:2109.00486v3cs.CL

TL;DR

Low-resource MT must produce useful models despite scarce parallel data, a condition affecting most language pairs and often compounded by limited evaluation coverage. This survey synthesizes methods for exploiting data, related languages, and model design across the field and shared tasks. It concludes that progress is possible for some pairs with moderate parallel data and substantial synthetic or multilingual resources, but many other pairs remain very poor or impossible to translate.

  • Problem

    Most language pairs have extremely limited or nonexistent parallel data, while standard evaluation evidence covers relatively few low-resource and non-English pairs.

  • Method

    The survey organizes low-resource MT research around data resources, monolingual and related-language transfer, multilingual models, and other techniques evaluated across shared tasks.

  • Results

    The survey finds a mixed picture: acceptable results can be obtained for some pairs with moderate parallel data, large-scale multilingual systems, and ample synthetic data, while many others remain very poor or impossible.

  • Takeaways & Limitations

    Web crawling, monolingual data, and related languages have helped some pairs move from low-resource to medium-resource status, although high-resource systems still require huge datasets.

  • Takeaways & Limitations

    Evaluation conclusions may not transfer reliably from high-resource languages, and FLORES-101’s non-English tests use translationese rather than mutual translations.

Abstract

from arXiv · show

We present a survey covering the state of the art in low-resource machine translation research. There are currently around 7000 languages spoken in the world and almost all language pairs lack significant resources for training machine translation models. There has been increasing interest in research addressing the challenge of producing useful translation models when very little translated training data is available. We present a summary of this topical research field and provide a description of the techniques evaluated by researchers in several recent shared tasks in low-resource MT.

1. Introduction

Low-resource MT addresses language pairs with extremely limited or nonexistent data, often compounded by noisy, out-of-domain corpora and missing linguistic tools. This survey organizes recent approaches and shared-task techniques for overcoming these constraints.

  • Motivation: Most language pairs have extremely limited or nonexistent training data, unlike highly resourced pairs with tens or hundreds of millions of parallel sentences.Highly resourced examples typically involve English paired with European languages, Arabic, or Chinese.
  • Motivation: Resource levels do not consistently follow speaker numbers: widely spoken languages may have little parallel data, while some small languages are richly resourced.Opus provides approximate parallel-sentence counts used to illustrate these differences.
  • Motivation: Available corpora may be noisy, specialized, or mismatched with the intended domain, including religious and IT-localization sources.Low-resource settings also commonly lack reliable linguistic tools such as language identification.
  • Scope and organization: The survey provides broad coverage of low-resource MT research, including tools and techniques validated across 18 shared tasks from 2018 to 2021.Its focus is mainly neural MT, and it is intended to help researchers and practitioners, especially newcomers, assimilate recent directions.
  • Scope and organization: Because resource availability changes over time and lies on a continuum, the survey distinguishes simulated low-resource settings from genuinely low-resource languages.The inclusion criterion is broad: research qualifies when it addresses the implications of scarce data or proposes ways to overcome them.
  • Scope and organization: The survey structures methods around increasing data quantity and quality, exploiting monolingual, cross-lingual, and linguistic resources, improving training, and building research capacity.The organization follows these method categories across the paper’s sections.

2. Data Sources

Low-resource MT relies on targeted collection, mining, and creation of parallel and monolingual data, while standardized benchmarks support evaluation. Web crawling can expand resources but is especially vulnerable to noisy alignment and unreliable language-processing tools.

  • Data collection: Carefully targeted data gathering can produce clear MT improvements, making resource identification and collection a central consideration for new language pairs.The survey discusses existing datasets and methods for extracting additional parallel data.
  • Existing resources: Opus aggregates parallel corpora from more than 500 languages and variants, while CommonCrawl-derived resources provide broad monolingual coverage after cleaning.Examples include Oscar, CC100, and mc49.
  • Web crawling: Web crawling extracts and sentence-aligns translated text, but broad efforts tend to favor high-resource languages; focused crawling can better boost low-resource pairs.Focused efforts may target particular language pairs or sources, including multilingual government documents.
  • Web crawling: Because parallel sentences are rare relative to web content, even a small false-positive rate can make a crawled corpus mostly noise.Errors include bad alignments, wrong-language sentences, and HTML or JavaScript fragments.
  • Web crawling: Low-resource crawling pipelines may lack reliable language identification, translation systems, sentence embeddings, and other preprocessing resources.Language identification is affected by class imbalance, similar languages, encoding problems, and domain mismatches.
  • Web crawling: Crawled sources should be treated cautiously because quality analyses find incorrect language identification, non-parallel sentences, low-quality text, and offensive language.These problems can be more acute for low-resource languages.
  • New data creation: Crowdsourcing offers a cost-effective way to create parallel corpora and bilingual dictionaries across multiple low-resource languages.Prior work applied it to several Indian languages and broad multilingual dictionary collection.
  • Test sets: Shared tasks provide useful standardized test sets, but coverage remains narrow and usually includes English; FLORES-101 expands multilingual and non-English evaluation.For non-English pairs, FLORES-101 uses translationese on both sides rather than mutual translations.

3. Use of monolingual data

Because monolingual text is usually more plentiful than parallel text, low-resource MT research widely exploits it through language-model integration and synthetic parallel data. Backtranslation dominates this area, while its variants and unsupervised MT extend the same basic strategy.

  • Overview: Monolingual data is usually more plentiful than parallel data and has become one of the most important and successful resources in low-resource MT.The survey reviews language-model integration, synthetic data, unsupervised MT, and modifications of existing parallel data.
  • Language-model integration: External language models can improve NMT through shallow fusion, which interpolates probabilities, or deep fusion, which integrates language-model hidden states into the decoder.These approaches improved scores across scenarios, including simulated low-resource Turkish→English.
  • Language-model integration: Using a language model as a training-time regularizer avoids adding it directly to decoding, and yielded considerable gains in simulated very low-resource settings.Reported experiments used small Turkish–English and German–English datasets.
  • Synthetic parallel data: Synthetic parallel data translates monolingual text with a heuristic or intermediate MT model, leaving one side human-generated and the other automatically produced.The survey focuses on backtranslation and related variants.
  • Synthetic parallel data: Backtranslation translates target-side monolingual data into synthetic source sentences so the decoder sees well-formed target-side text.It became dominant because of reported improvements and its relevance to unsupervised MT.
  • Synthetic parallel data: In simulated low-resource settings, beam search can outperform sampling for backtranslation when data is very scarce, whereas sampling is favored with more data.This pattern is reported by Edunov et al. and appears confirmed by Xu et al.
  • Synthetic parallel data: Forward translation, target-to-source copying, and iterative backtranslation provide alternatives or extensions, with copying especially useful in tested low-resource settings.Iterative backtranslation addresses poor initial models by using intermediate models of increasing quality.
  • Unsupervised MT: Unsupervised MT aims to learn translation without parallel data through rough cross-lingual initialization followed by iterative backtranslation and denoising autoencoding.It represents an extreme form of low-resource MT, though negative results have motivated further improvements.

4. Use of multilingual data

Multilingual data helps low-resource MT through transfer learning and multilingual models, but gains depend on vocabulary overlap, model capacity, sampling, and the balance between transfer and interference.

  • Transfer learning: Transfer learning initializes a low-resource child model with parameters from a parent model trained on another language pair.The parent model can be one-to-one, while later methods also use multi-stage transfer through an intermediate multilingual model.
  • Transfer learning: Sharing vocabularies benefits transfer between related languages, while BPE, embedding mapping, or transliteration can increase overlap when languages or scripts differ.Externally pretrained embeddings can also be fixed during parent training before initializing the low-resource model.
  • Transfer learning: Strong parent models and data-related features often matter more than language similarity, although unrelated languages with different scripts can reduce transfer effectiveness.Studies disagree on the importance of parent-language identity, with some finding it important and others finding it unimportant.
  • Multilingual models: Multilingual models exploit data from multiple languages through shared representations and parameters, including universal encoder-decoder architectures with joint vocabularies and target-language tokens.Multilingual training can also enable zero-shot translation between language directions without direct parallel data.
  • Massively multilingual models: Multi-task multilingual models outperform bilingual models in the reported many-to-many experiments, while larger language sets improve zero-shot translation.The many-to-many models shared attention across directions, increasing parameters linearly rather than quadratically.
  • Massively multilingual models: Temperature-based sampling balances low-resource representation against deterioration on high-resource language pairs caused by upsampling.Larger models improve translation broadly; deeper models are particularly successful for low-resource languages, whereas wider models are more prone to overfitting.
  • Large-scale multilingual pre-training: Large-scale multilingual pre-training with noisy autoencoders improves supervised, unsupervised, and semi-supervised NMT, including low-resource language pairs.mBART demonstrated unsupervised NMT for Nepali–English and Sinhala–English and supervised improvements for Gujarati–English.
  • Large-scale multilingual pre-training: Pre-trained models are constrained by language coverage and the resources required to train and deploy very large models.Distant languages and different scripts create additional difficulties, although careful transliteration can mitigate script differences.

5. Use of external resources and linguistic information

The survey examines how external resources and linguistic information can compensate for sparse low-resource MT data. Approaches include subword segmentation, factored and multitask models, interleaving, syntactic reordering, and bilingual lexicons.

  • 5.1.1 Morphological segmentation: Subword segmentation represents an infinite vocabulary with fixed units and improves generalisation over shorter units, but segmentation choices produce variable low-resource results.BPE and SentencePiece are common statistics-based strategies, and the segmentation degree must be selected carefully.
  • 5.1.1 Morphological segmentation: Morphologically aware segmentation found no gain over BPE for Nepali, Sinhala, and Kazakh, with no consistent pattern across results.These findings contrast with earlier Turkish results but agree with more recent Kazakh–English and Tamil–English studies.
  • 5.1.2 Factored models: Factored representations decompose words into components such as lemmas and part-of-speech tags, with improvements depending on the language pair and included linguistic information.NMT factors are represented in continuous space, making them easier to combine than in earlier SMT factored models.
  • 5.1.3 Multi-task learning: Multi-task learning trains translation alongside linguistic analyses to encourage better internal wordform representations, with reported slight gains in lower-resource settings.Auxiliary tasks include parsing, named entity recognition, semantic parsing, and syntactic parsing; task interference remains a training concern.
  • 5.1.4 Interleaving of linguistic information: Interleaving linguistic annotations benefits translation across 8 language directions, with source-side morphology and target-side part-of-speech tags showing differing advantages.Target-side morphological information produced more grammatical translations, although target-side PoS tagging was more effective for translation performance.
  • 5.1.5 Syntactic Reordering: Syntactic reordering produced significant translation-quality gains when transferring from English→Hindi and reordering Bengali, Gujarati, Marathi, Malayalam, and Tamil sources.The method applies source-side rules to match the child source language's syntactic order.
  • 5.2 Bilingual lexicons: Bilingual lexicons supply specialist or unseen vocabulary, and pretrained multilingual use of lexicon substitutions shows the most promise among recent lexicon approaches.The substitutions aim to bring words with similar meanings across languages closer in representation space, while lexicons lack contextual translation information.

6. Model-centric Techniques

Model-centric techniques seek to improve low-resource MT when limited data makes model inductive bias more prominent, especially outside the training distribution. The survey covers meta-learning, latent-variable models, alternative objectives, inference methods, and rule-based translation.

  • 6. Model-centric Techniques: Low-resource MT makes model inductive bias more prominent, particularly when systems operate outside the training distribution.The survey therefore reviews methods for better adaptation, uncertainty modelling, training objectives, and inference.
  • 6.1 Meta Learning: Meta-learning trains models to adapt quickly to novel language-pair tasks using few examples, and MAML applied to multilingual MT obtained substantial improvements on true low-resource pairs.Gu et al. meta-trained on synthetic tasks created by subsampling high-resource parallel corpora before fine-tuning on true low-resource pairs.
  • 6.2 Latent variable models: Latent-variable models increase expressivity and allow more flexible inductive-bias design, but require more complicated and computationally expensive training and inference.Approximation methods such as Monte Carlo sampling and MAP inference are typically used within VAE-based approaches.
  • 6.3 Alternative training objectives: Alternative objectives expose models during training to sequences generated by themselves to address exposure bias, but some reported improvements come from simulated or relatively small-resource settings.MIXER improved German→English on IWSLT 2014 but was not evaluated on a true low-resource language pair; CMRT improved out-of-domain German→Romansh and reduced hallucinations.
  • 6.4 Alternative inference algorithms: Increasing beam size can improve MAP-inference approximation while degrading translation quality beyond a certain point, motivating alternative inference algorithms.Ancestral sampling was theoretically less biased but performed worse than both beam search and MBR in the reported comparison.
  • 6.4 Alternative inference algorithms: Energy-based models rerank autoregressive NMT samples using a reference-based metric, approximating MAP inference over a distribution that avoids high weight for abnormal translations.The model requires a proposal distribution to generate initial samples because its energies are unnormalised.
  • 6.5 Rule-based approaches: Rule-based MT requires substantial language-specific resources and expertise, which may be unavailable for low-resource languages.This resource requirement is a central scope boundary for applying RBMT in low-resource settings.

7. Evaluation

Low-resource MT evaluation must balance human assessment with automatic metrics, despite limited references, unreliable test sets, and scarce evaluators. The survey recommends human evaluation as the gold standard and caution with automatic measures.

  • Evaluation approaches: Human evaluation is the gold standard, while automatic metrics remain useful but imperfect substitutes for human assessment.Evaluation priorities depend on intended use: adequacy matters for gisting, whereas fluency and publishability matter for dissemination.
  • Human evaluation: Low-resource evaluation is constrained by small evaluator pools and limited connections between researchers and language communities.The survey suggests greater engagement between researchers and language communities when evaluator access is limited.
  • Evaluation limitations: WMT metrics shared-task evidence covers mainly higher-resource language pairs, so its conclusions may not transfer reliably to low-resource languages.The survey therefore warns against assuming that metric utility validated on WMT news languages generalizes to low-resource settings.
  • Automatic evaluation: Automatic metrics require high-quality reference translations, but low-resource language pairs often lack references or have references from the wrong domain.Available references may also be small or poor quality, weakening reference-based evaluation.
  • Automatic evaluation: Embedding-based metrics generally correlate better with human judgements than string-based metrics, but depend on sentence embeddings and sometimes human fine-tuning data.String-based metrics require fewer resources, whereas embedding-based metrics may be unavailable or less reliable without suitable training data.
  • Recommendations: The survey recommends human evaluation as the gold standard and warns about metric calibration and test-set reliability in low-resource MT.These concerns apply especially when automatic evaluation is used in place of human judgements.

8. Shared Tasks

Shared tasks standardize data and evaluation, enabling reproducible comparisons of low-resource MT systems and revealing recurring successful practices. High-performing submissions commonly combine data augmentation, multilingual or related-language resources, model ensembles, and tuned preprocessing choices.

  • Shared-task role: Shared tasks keep training and testing data constant, allowing researchers to reproduce and combine methods across systems.They provide a common setting for assessing whether research ideas transfer beyond individual published experiments.
  • Shared-task role: System descriptions show how technique combinations perform under standardized conditions, but their pipeline structure makes individual contributions difficult to isolate.Leading and weaker systems may share similar techniques, and descriptions are not always exhaustive.
  • Shared-task role: Published shared-task training and test sets support later research and raise the profile of low-resource language pairs.These datasets are an important reusable output beyond the competition itself.
  • Survey organization: The survey reviews low-resource shared-task choices step by step, from data preparation and processing through architecture and additional-data exploitation.Its focus is on widespread, effective, and language- or family-specific techniques, especially in high-performing systems.
  • Data choices: Most constrained submissions use all available data, while teams vary in whether they omit or quality-filter web-crawled material.Web-crawled data is treated cautiously except for the generally more homogeneous WMT newscrawl.
  • Data processing: 30k-60k merge operations are common despite low-resource settings, whereas many Tamil–English systems use 6k-30k merge operations.Subword vocabulary size varies substantially and does not show a consistent pattern across systems.
  • Data processing: Morphology-driven segmentation has not shown a consistent advantage over BPE across Upper-Sorbian–German and Tamil–English studies.Reported findings range from parity to gains, depending on language pair and tuning.
  • Model architecture: NMT, especially transformer models, has overtaken SMT in low-resource shared tasks when sufficient data, including synthetic data, is available.Some teams still use SMT for backtranslation because it can perform better or be less susceptible in particular settings.

9. Conclusion

The survey finds mixed progress in low-resource machine translation: data collection, multilingual models, monolingual data, and related languages have improved some language pairs, but translation quality remains uneven. It synthesizes current practices and identifies collaboration, external knowledge, robustness, and multilingual-model analysis as priorities for future work.

  • Quantifiable results: Shared tasks rarely provide definitive answers but indicate which combinations of techniques can produce state-of-the-art systems for particular language pairs.The survey uses these practical patterns to provide perspective on the research covered.
  • Survey scope: The survey covers the field from data sourcing and multilingual or monolingual data to linguistic resources, model robustness, training, inference, and evaluation.It presents this as the full spectrum of scientific effort examined across Sections 2–7.
  • Progress: Large-scale web data collection and learning from monolingual and related-language data have moved some language pairs from low-resource toward medium-resource status.Examples include Romanian–English, Turkish–English, and Hindi–English.
  • Quantifiable results: Recent comparisons show a mixed performance gap between high- and low-resource language pairs, while acceptable results may be possible with moderate parallel data, multilingual systems, and ample synthetic data.The comparison between German–English and Hausa–English illustrates the difficulty of comparing results across language pairs.
  • Future work: Future work should involve language communities, investigate massive multilingual models, incorporate external knowledge, and improve robustness to changing domains, topics, and registers.The survey emphasizes collaboration with speakers, unresolved questions about multilingual-model gains and knowledge encoding, and degradation under distribution shift.
  • Conclusion: The survey aims to give researchers a broad understanding of low-resource machine translation so they can develop more effective tools and resources.Its stated scope is a challenging but essential research field.

2021. Findings of the wmt 2021 shared

This passage set consists primarily of references to prior work and shared-task reports related to low-resource and multilingual machine translation. The cited work spans methods, evaluation, language coverage, and applications.

  • The references include studies of unsupervised and very-low-resource supervised machine translation.
  • Other cited studies address evaluation metrics, robustness, word-order divergence, linguistic prediction, and training-data selection.
  • The cited literature covers multilingual pretraining, alignment, transfer languages, and massively multilingual neural MT.
  • Several references report shared-task findings for Asian, Indigenous American, and very-low-resource translation settings.
Loading 2109.00486v3…