PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Machine Translation for Low-Resource Languages: A Survey

Surangika Ranathunga, En-Shiun Annie Lee, Marjana Prifti Skenduli, Ravi Shekhar, Mehreen Alam, Rishemjit Kaur

arXiv:2106.15115v1cs.CLcs.AI

TL;DR

Low-resource language pairs lack the large parallel corpora that NMT typically requires, leaving their performance sub-optimal. This survey systematically reviews LRL-NMT techniques and analyzes the research landscape quantitatively. It provides technique-selection guidelines and recommends resources, public models, computational access, and regional community involvement, while noting limited study and coverage for many LRLs.

  • Problem

    NMT performance on low-resource language pairs remains sub-optimal because large parallel corpora are unavailable.

  • Method

    The paper systematically reviews LRL-NMT techniques, analyzes their practical applicability, and conducts a quantitative analysis of the research landscape.

  • Results

    The survey provides guidelines for selecting NMT techniques for given LRL data settings and identifies factors associated with research growth across languages.

  • Takeaways & Limitations

    Advancing LRL-NMT requires LRL resources, publicly available computational resources and trained models, and regional research-community involvement.

  • Takeaways & Limitations

    Unsupervised NMT remains insufficiently studied for extreme LRLs, and massive multi-NMT models cover only a small fraction of more than 7000 languages.

Abstract

from arXiv · show

Neural Machine Translation (NMT) has seen a tremendous spurt of growth in less than ten years, and has already entered a mature phase. While considered as the most widely used solution for Machine Translation, its performance on low-resource language pairs still remains sub-optimal compared to the high-resource counterparts, due to the unavailability of large parallel corpora. Therefore, the implementation of NMT techniques for low-resource language pairs has been receiving the spotlight in the recent NMT research arena, thus leading to a substantial amount of research reported on this topic. This paper presents a detailed survey of research advancements in low-resource language NMT (LRL-NMT), along with a quantitative analysis aimed at identifying the most popular solutions. Based on our findings from reviewing previous work, this survey paper provides a set of guidelines to select the possible NMT technique for a given LRL data setting. It also presents a holistic view of the LRL-NMT research landscape and provides a list of recommendations to further enhance the research efforts on LRL-NMT.

1 INTRODUCTION

NMT requires large parallel corpora, creating a major challenge for low-resource languages despite their practical translation needs. This survey systematically reviews LRL-NMT techniques, analyzes their applicability, and develops selection guidelines and recommendations.

  • NMT systems require huge volumes of parallel data to achieve results comparable to SMT.
  • Large parallel-data requirements are unrealistic for many of the 7000+ languages currently in use, making LRL translation a major challenge.
  • Automatically translating LRLs is useful for countries with multiple official languages, motivating increased research on LRL language pairs.
  • Existing reviews did not comprehensively examine NMT techniques for LRLs, making technique selection difficult for a given dataset specification.
  • The survey systematically analyzes LRL-NMT techniques, evaluates practical applicability, and provides selection guidelines based on dataset characteristics and computing resources.
  • It recommends creating LRL resources, publicly sharing computational resources and trained models, involving regional research communities, and improving existing techniques.

2 BACKGROUND

The paper defines low-resource language status primarily through limited linguistic and electronic resources, while distinguishing language-level scarcity from domain-level scarcity. It positions itself as the first comprehensive survey focused on NMT for LRLs and excludes adjacent areas outside that scope.

  • 2.1 Low-Resource Languages (LRLs): An LRL may lack a unique writing system, web presence, linguistic expertise, or electronic resources such as monolingual and parallel corpora.
  • 2.1 Low-Resource Languages (LRLs): Joshi et al. categorised 2485 languages into six groups based on the amount of publicly available data.
  • 2.1 Low-Resource Languages (LRLs): In machine translation, a language pair’s resourcefulness is determined by the available parallel corpora, but no minimum corpus size defines resource categories.
  • 2.2 Related Work: Earlier surveys covered LRL-NMT briefly or focused on individual methodologies, rather than providing a comprehensive survey of NMT for LRLs.
  • 2.1 Low-Resource Languages (LRLs): Low-resource domains can occur in high-resource languages, whereas this paper focuses specifically on low-resource languages.
  • 2.2 Related Work: The review focuses on LRL translation and omits general NMT, speech-only, multimodal, and unrelated SMT research.

3 NMT TECHNIQUES FOR LOW-RESOURCE LANGUAGES

The survey organizes LRL-NMT methods around the availability of parallel and monolingual data, multilinguality, transfer, and zero-shot settings. These techniques adapt standard NMT or exploit shared models and related language resources.

  • 3.1 Overview: NMT methodologies broadly comprise supervised, semi-supervised, and unsupervised approaches.
  • 3.1 Overview: Supervised and semi-supervised NMT rely on parallel data, while semi-supervised methods additionally combine small parallel corpora with monolingual data.
  • 3.5 Multilingual NMT: Multi-NMT uses one model for more than one language pair because separate bilingual models for every pair are impractical.
  • 3.6 Transfer Learning in NMT: Transfer learning initializes an LRL child model from a model trained on a high-resource language pair, reducing training time and outperforming training from scratch.
  • 3.7 Zero-shot NMT: Zero-shot NMT addresses language pairs without parallel training data, with pivoting, multi-NMT, unsupervised NMT, and transfer learning identified as solutions.
  • 3.1 Overview: Figure 1 summarizes the techniques, while noting that semi-supervised NMT and multi-NMT support additional scenarios beyond those depicted.

3.2 Data Augmentation Techniques

Data augmentation expands training resources by generating synthetic parallel data through replacement, back-translation, or parallel-data mining. These methods can help LRL translation, but their usefulness depends on available models, resources, data quality, and language or domain relatedness.

  • Data augmentation generates synthetic parallel sentence pairs without usually changing the NMT architecture.
  • NMT data augmentation comprises word or phrase replacement, back-translation, and parallel corpus mining.
  • Word or phrase replacement creates synthetic sentences by substituting words or phrases using resources such as bilingual dictionaries.
  • Back-translation translates target-side monolingual data into the source language to construct synthetic parallel corpora.
  • Back-translated data is noisier than original parallel data when the generating MT system is suboptimal, motivating selection, filtering, and iterative back-translation.
  • Parallel-data mining extracts translation-equivalent sentences from comparable corpora such as Wikipedia or news articles covering the same facts.
  • Multilingual sentence embeddings identify candidate parallel sentences by ranking nearest neighbours across languages.
  • LRL augmentation is constrained because back-translation requires an existing MT system, replacement methods require language-specific resources, and success depends on data or language relatedness.

3.3 Unsupervised NMT

Unsupervised NMT reduces reliance on parallel data by training with monolingual corpora or cross-lingual representations. Its pipeline initializes shared representations, uses back-translation, and applies adversarial training, but resource and language/domain mismatches constrain LRL applicability.

  • Unsupervised NMT uses monolingual corpora or cross-lingual word embeddings, making it less data-intensive for low-resource translation.
  • Initialization: Initialization maps embeddings from two or more languages into a shared space using linguistic resources and neural input representations.The accompanying figure describes dictionary-guided rotation and alignment of language embeddings.
  • Back-Translation: Back-translation uses a denoising autoencoder to translate source-to-target and back, comparing the original text with the doubly translated output.
  • Adversarial Architecture: Adversarial training uses a binary classifier and an adversarial loss that trades off discrimination against reconstruction.The passage reports that this produces more fluent translation for LRLs.
  • LRL unsupervised NMT is limited by missing dictionaries or alignments, large monolingual-data requirements, language or domain dissimilarity, and extreme low-resource conditions.Reduced quality has been reported for language pairs from dissimilar linguistic families or domains.

3.4 Semi-Supervised NMT

Semi-supervised NMT combines some parallel data with monolingual data through synthetic-data generation, language modeling, objective changes, multitask learning, or dual learning. Its applicability to true LRL settings remains uncertain because evaluations often use simulated scarcity, very large monolingual corpora, and limited comparisons.

  • Semi-supervised NMT assumes some parallel corpora alongside monolingual corpora and categorizes methods by how monolingual data is used.
  • Synthetic Parallel Data: Monolingual data can generate synthetic parallel data by copying text, using null source tokens, or applying back-translation.
  • Language Models: Language models support NMT through shallow or deep fusion, weakly informative priors, or parameter initialization.
  • Other approaches incorporate monolingual data through reconstruction objectives, multitask learning, or dual learning with iterative two-hop translation.Dual learning evaluates reconstructed sentences against original monolingual inputs to produce reward signals.
  • Evidence for LRL applicability is constrained because studies simulate scarcity with high-resource languages, assume very large monolingual corpora, and rarely compare methods broadly.Some reported results are below or only marginally above back-translation and iterative back-translation.

3.5 Multilingual NMT

Multilingual NMT handles multiple language pairs through several architectural and task configurations, with parameter sharing and input representation central to performance. Results are promising for LRLs, but language imbalance, noise, and linguistic or topical discrepancies remain challenges.

  • Multilingual NMT translates among more than one language pair, and models with roughly 50 languages have shown gains over bilingual models for LRLs in English-centric datasets.
  • Challenges: Multi-NMT for LRLs faces varying language differences, noisy parallel data, data imbalance, and discrepancies in writing style or topic.
  • Translation Tasks: The main task configurations are one-to-many, many-to-one, and many-to-many, with many-to-many described as the most difficult scenario.
  • Architectures: Supervised architectures include a universal shared encoder-decoder, separate per-language encoder-decoders, and partially shared encoder-decoder arrangements.The universal model supports one-to-many, many-to-one, and many-to-many translation while reducing model complexity.
  • Architecture Trade-offs: Architecture comparisons favor language-specific decoders or attention and partial parameter sharing, while maximum sharing is desirable for hundreds of languages to reduce complexity.
  • Input Representation: Input representation determines whether semantically similar words from different languages can be grouped in the embedding space.Representations are broadly divided into surface-form and embedding-based categories.
  • Transfer learning over multi-NMT models has produced promising results, while research increasingly uses actual LRLs but often overlooks individual language and family characteristics.Unsupervised multilingual models can also improve when at least one language has parallel data with another language.

3.6 Transfer Learning in NMT

Transfer learning initializes an LRL child model from a parent model trained on high-resource parallel data, with choices concerning fine-tuning, transfer protocols, and language-space alignment. It consistently outperforms training from scratch in reported LRL settings, but evidence for newer large multilingual parents remains limited.

  • Transfer learning initializes an LRL child model with parameters from a parent model trained on a larger high-resource parallel corpus.
  • Reported benefits include reduced child-data requirements, improved child-task performance, and faster convergence than training from scratch.
  • Transfer Setting: Transfer systems may use warm-start or cold-start protocols; warm-start systems are more accurate because child parallel data is available during parent training.
  • Transfer-learning improvements address language-space mismatch, fine-tuning technique, and transfer protocol.Mismatch may appear as vocabulary mismatch when surface-form inputs are used.
  • Fine-Tuning: Fine-tuning ranges from freezing the whole parent model to fine-tuning selected layers or all layers, with the best setup depending on architecture and task.
  • Transfer Protocol: Transfer protocols include parameter chains, combined parent-child training followed by child-only fine-tuning, and staged transfer through an intermediate related language.
  • Results and Scope: High-resource-parent transfer consistently outperforms training LRL child models from scratch, including extremely low-resource children.Some studies also report transfer outperforming earlier multi-NMT training, but those comparisons used few languages and an early model.
  • Limitations: Evidence remains limited for transfer from newer multi-NMT parents with many language pairs, and the effects of relevant factors require further evaluation.Zero-shot translation adds the condition that child parallel data is unavailable.

3.7 Zero-shot NMT

Zero-shot LRL translation is addressed through pivoting, transfer learning, multi-NMT, and unsupervised NMT, with each approach relying on different data and modeling assumptions. The survey highlights both performance gains and practical limitations across these techniques.

  • Four solution families are identified for zero-shot NMT: pivoting, transfer learning, multi-NMT, and unsupervised NMT.
  • Pivot-based solutions: Pivoting decomposes source-target translation into independently trained source-pivot and pivot-target models, but can propagate errors and increase training time.
  • Pivot-based solutions: Small amounts of true source-target parallel data increase pivoting accuracy, while pivot-language monolingual data can support synthetic parallel-data generation.
  • Multi-NMT-based solutions: Multi-NMT models can outperform pivot-based zero-shot translation, and many-to-many models can outperform English-centric models in zero-shot settings.
  • Multi-NMT-based solutions: Multi-NMT can learn an interlingua from multilingual parallel data, but model-capacity bottlenecks prevent that representation from being fully language independent.
  • Unsupervised NMT-based solutions: Unsupervised NMT relies only on monolingual data, making it a zero-shot translation approach.

3.8 Analysis on the Popularity of LRL-NMT Techniques

The survey quantifies LRL-NMT research interest using Google Scholar results from 2014–2020, while acknowledging search-result noise. Multi-NMT led through 2019, but unsupervised techniques marginally surpassed it afterward, alongside rising interest in several other techniques.

  • Google Scholar searches compare the popularity of seven LRL-NMT techniques over 2014–2020, with acknowledged noise assumed to be equally distributed.
  • Multi-NMT had the highest number of papers through 2019, while unsupervised techniques marginally surpassed it afterward.
  • Transfer learning increased steeply from 2019 onward, and data augmentation gained popularity in 2020.
  • Pivoting lost traction, possibly following advances in multi-NMT that outperformed pivoting for zero-shot translation.
  • Overall, LRL-NMT research interest steadily increased across technique types.

4 GUIDELINES TO SELECT A TECHNIQUE FOR A GIVEN DATA SPECIFICATION

The survey presents advisory guidelines for choosing LRL-NMT techniques according to parallel and monolingual data, language similarity, and computing resources. It frames selection as iterative: practitioners can augment, train, generate, clean, and reintegrate data as resources develop.

  • Technique effectiveness and viability depend on parallel and monolingual data characteristics and available computational resources.
  • The selection flowchart primarily covers bilingual settings with parallel data, while multilingual techniques remain possible when sufficient computing resources are available.
  • The main selection factors are parallel-corpus availability and size, monolingual-corpus availability, and language similarity.
  • With fewer than 0.5M parallel sentences, practitioners can augment data before applying supervised, semi-supervised, or related techniques.
  • When source and target languages share parallel data with a common high-resource language, pivoting is recommended; otherwise, transfer learning can use a high-resource parent model.
  • Without parallel data but with reasonable monolingual corpora, unsupervised NMT can be applied; without either resource, corpora must be manually created.
  • LRL-NMT is described as iterative, with generated parallel data cleaned and reintegrated to train new models.

5 LANDSCAPE OF LOW-RESOURCE LANGUAGES AND NMT RESEARCH

The survey examines which low-resource languages receive NMT research attention and relates that attention to geography, dataset availability, and access to open tools and community projects.

  • Methodology: The analysis queried Google Scholar results for languages in Classes 0–2, excluding results before 2014, patents, and citations.The authors note that language-name ambiguity made the search results noisy.
  • Research activity by language class: 12.6%, 11.2% and 7.1% of languages were identified as outliers in Classes 0, 1 and 2, respectively.Examples include Sinhala and Slovene in Class 0, Nepali and Telugu in Class 1, and Irish in Class 2.
  • Geographic considerations: Approximately 25% of Class 0 outliers were European languages, compared with approximately 7% of all Class 1 European languages being outliers.The authors associate early NMT growth with geographic location and suggest funding, resources, and regional projects as possible contributors.
  • Dataset availability: Sinhala and Nepali showed a steep rise in research activity from 2018–19 onwards, potentially following the release of FLoRes datasets and inclusion in WMT challenges.The cited passage identifies datasets and yearly challenges as examples of factors associated with language-specific research growth.
  • Dataset availability: The number of available datasets was directly correlated with research activity, with r=0.88 across a resource matrix covering 64 languages.The resource matrix included monolingual and parallel corpora, though the authors note that its language list was not exhaustive.
  • Open resources and community involvement: Open-source frameworks, pretrained models, and regional communities support LRL-NMT growth by lowering implementation barriers and developing datasets, tools, and expertise.The survey highlights OpenNMT, fairseq, mBART, and the Masakhane project as examples.

6 DISCUSSION

The discussion identifies model improvements and equitable access as priorities for advancing LRL-NMT. It emphasizes broader language coverage, robustness, interpretability, bias mitigation, and expanded access to datasets, tools, models, and computing resources.

  • Model improvements: The survey proposes improving multilingual coverage, robustness to limited inputs, interpretability, explainability, and bias mitigation for LRL-NMT models.These priorities address limitations involving language coverage, data conditions, model behavior, and dataset bias.
  • Model capacity to include LRLs: Massive multilingual models can cover about 100 languages, but only a small fraction of more than 7000 possible languages are currently included.The authors call for scaling multilingual models while preserving high-resource-language performance.
  • Model robustness: LRL-NMT techniques remain constrained by dataset size, domain, and language relatedness, while existing ablation studies are not sufficient or exhaustive.The discussion identifies spoken–written syntactic differences, code-mixed data, and non-realistic dataset domains as challenges.
  • Equitable and inclusive access: The survey argues that underrepresented geographic regions should receive more resources, including datasets and substantial computing access.This recommendation follows the trend analysis and focuses on communities excluded from technological development or facing socioeconomic inequities.
  • Resources and infrastructure: The survey highlights the need for more LRL datasets, open-source tools and frameworks, publicly released trained models, and distributed computational resources.Public models could serve as parent models for transfer to LRL child models, while conference- and challenge-based resources could encourage participation.
  • Key findings: All reviewed LRL-NMT techniques except pivoting showed upward publication trends, while the decision chart guides technique selection using data specifications and computational resources.The authors note that technique selection also depends on language relatedness and data domain.

7 CONCLUSIONS

The survey consolidates the LRL-NMT landscape and offers guidance for choosing techniques by data specification. It also uses literature findings and quantitative analysis to recommend ways to advance LRL-NMT solutions.

  • The survey provides a comprehensive picture of LRL-NMT research and highlights recent technological trends.
  • It gives guidelines for selecting an appropriate LRL-NMT technique for a given data specification.
  • Its recommendations are based on findings from research publications and quantitative analysis.
Loading 2106.15115v1…