Source-linked AI summary
A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios
Michael A. Hedderich, Lukas Lange, Heike Adel, Jannik Strötgen, Dietrich Klakow
TL;DR
Low-resource NLP must address sparse labeled data across languages, domains, and tasks, while available auxiliary resources differ substantially. This survey organizes the dimensions of resource availability and reviews data augmentation, distant supervision, and transfer learning methods by their assumptions and requirements. It concludes that comparing approaches across resource conditions remains important, while surveyed techniques have meaningful coverage and suitability boundaries.
Problem
Low-resource NLP lacks sufficient labeled data, and suitable techniques depend on the available language, domain, and auxiliary resources.
Method
The survey analyzes resource-availability dimensions and structures approaches including data augmentation, distant supervision, and transfer learning by their requirements and assumptions.
Results
The survey identifies resource availability as a spectrum and emphasizes that techniques differ in applicability across low-resource settings.
Takeaways & Limitations
Practitioners should analyze a setting’s data dimensions and auxiliary resources when choosing among low-resource NLP techniques.
Takeaways & Limitations
Multilingual models do not provide universal language coverage, and low-resource African and American languages are particularly underrepresented.
Abstract
from arXiv · showhide
Deep neural networks and huge language models are becoming omnipresent in natural language applications. As they are known for requiring large amounts of training data, there is a growing body of work to improve the performance in low-resource settings. Motivated by the recent fundamental changes towards neural models and the popular pre-train and fine-tune paradigm, we survey promising approaches for low-resource natural language processing. After a discussion about the different dimensions of data availability, we give a structured overview of methods that enable learning when training data is sparse. This includes mechanisms to create additional labeled data like data augmentation and distant supervision as well as transfer learning settings that reduce the need for target supervision. A goal of our survey is to explain how these methods differ in their requirements as understanding them is essential for choosing a technique suited for a specific low-resource setting. Further key aspects of this work are to highlight open issues and to outline promising directions for future research.
1 Introduction
Low-resource NLP addresses the lack of training data affecting thousands of languages and many domains or tasks. This survey structures existing methods by their resource requirements and assumptions to guide technique selection.
- NLP research focuses heavily on 10 to 20 high-resource languages, leaving thousands of languages with billions of speakers underrepresented.
- Low-resource scenarios span threatened languages, widely spoken languages with limited NLP coverage, and data-scarce domains or tasks in otherwise high-resource languages.Yongning Na is described as having 40k speakers and only 3k written, unlabeled sentences.
- Resource-lean methods overcome limited labeled data by leveraging sources such as unlabeled data, manual heuristics, and cross-lingual alignments.
- The survey provides a structured overview, analyzes resource dimensions, highlights data assumptions for practitioners, and discusses open issues and future directions.Table 1 summarizes surveyed techniques and their requirements.
2 Related Surveys
This paper focuses on general methods for low-resource supervised NLP rather than topics already covered by specialized surveys.
- The survey excludes low-resource machine translation and unsupervised domain adaptation because recent surveys already cover those topics.
- Its focus is general supervised methods, including data augmentation, distant supervision, and transfer learning.
3 Aspects of “Low-Resource”
Low-resource NLP is a spectrum defined by task labels, unlabeled text, and auxiliary data rather than a fixed threshold. Resource availability varies across languages, tasks, and data sizes, affecting which techniques apply.
- Resource coverage: Low-resource languages generally lack the data foundation for higher-level applications and often have smaller or lower-quality training data than English.
- Resource coverage: Figure 1 compares which NLP tasks are supported across six languages, but does not measure data quality or system performance.The tasks range from tokenization to question answering.
- Resource coverage: Four American and African languages with 1.5 to 60 million speakers have received less attention than Estonian, which has 1 million speakers.
- Dimensions of resource availability: Low-resource settings should be categorized by task-specific labels, unlabeled language- or domain-specific text, and auxiliary data.Auxiliary data may include labels in another language or domain, knowledge bases, gazetteers, or machine translation tools.
- How Low is Low-Resource?: There is no hard threshold for low-resource settings because requirements depend on the task and language.Reported examples range from 1–2k POS-tagged tokens to 350k labeled text-generation instances.
- How Low is Low-Resource?: For very small datasets, non-neural methods can outperform modern approaches, while neural methods perform better once a few hundred labeled instances are available.
4 Generating Additional Labeled Data
The survey groups ways to create labeled data when task-specific supervision is scarce, including augmentation, distant supervision, cross-lingual projection, and noisy-label handling. These approaches differ in their reliance on linguistic expertise, auxiliary resources, annotation rules, parallel data, or label-noise models.
- Data Augmentation: Data augmentation creates additional task-specific instances by transforming existing examples without changing their labels.Transformations can replace tokens, manipulate sentence structure, paraphrase through back-translation, generate label-conditioned text, or apply meaning-preserving perturbations.
- Open Issues: Data-generation methods face practical limits because augmentation lacks a unified cross-task, cross-language framework, while distant supervision depends on suitable auxiliary data and human-designed annotation techniques.Distant supervision can require substantial expert effort, and its suitability beyond information extraction remains an open question.
- Open Issues: The survey emphasizes using linguistic or domain expertise when unlabeled data or hardware resources are limited.Its broader framing treats resource assumptions as essential for selecting methods suited to particular low-resource settings.
- Distant and Weak Supervision: Distant supervision labels unmodified text automatically using external information such as dictionaries, gazetteers, metadata, heuristics, or other classifiers.It is widely used for named entity recognition and relation extraction, but has also been applied to part-of-speech tagging, aspect, discourse, topic, and text classification.
- Cross-Lingual Projection: Cross-lingual projection transfers labels from a classifier trained in a high-resource language through aligned parallel texts into a low-resource language.The approach requires both labeled high-resource data and reliable alignment or machine translation resources.
- Learning with Noisy Labels: Learning with noisy labels can filter probably incorrect instances or model the relationship between noisy and unseen clean label distributions.Filtering may remove examples or reweight them, while noise modeling commonly uses a confusion matrix appended to the classifier.
5 Transfer Learning
Transfer learning reduces the need for labeled target data by reusing representations and models learned from unlabeled or higher-resource data. The survey covers pretrained language representations, domain adaptation, multilingual transfer, and alignment, while noting that current models remain unevenly suited to real low-resource settings.
- Overview: Transfer learning reduces labeled target-data requirements by transferring learned representations and models, especially through language models pretrained on unlabeled data.The section uses BERT as a prominent example of pretrained language representations.
- Pretrained Representations: Subword embeddings address out-of-vocabulary problems and can outperform word-level embeddings on low-resource sequence-labeling tasks.The survey discusses n-gram and byte-pair-encoding embeddings as subword-based alternatives.
- Open Issues: Current pretrained and multilingual models remain constrained by hardware demands, limited data quality, domain gaps, and incomplete or uneven language coverage.mBERT covers 104 languages and XLM-R 100, while African and American language families are particularly underrepresented; distant families are less suited for transfer.
- Domain-Specific Pre-Training: Domain-adaptive and task-adaptive pretraining with unlabeled data improves performance across numerous high- and low-resource domains and tasks.Domain-specific models such as SciBERT can outperform general-domain BERT in specialized settings.
- Domain-Specific Pre-Training: Domain transfer can combine high-resource general-domain embeddings with low-resource target-domain embeddings or align diverse embedding spaces into domain-invariant representations.Attention-based meta-embeddings create weighted combinations, while adversarial alignment is used to reduce domain-specific differences.
- Multilingual Language Models: Multilingual models enable cross-lingual transfer by combining monolingual representations or training one model across languages, including zero-shot application to unseen target languages.Cross-lingual zero-shot learning trains on labeled data in a high-resource language and applies the multilingual model to the low-resource language.
- Multilingual Language Models: Embedding alignment maps different language spaces so corresponding words share similar feature vectors, allowing multilingual models to use compatible representations.This alignment supports transfer across languages and can address languages occupying separate spaces within a model.
6 Ideas From Low-Resource Machine Learning in Non-NLP Communities
The survey draws on machine learning and computer vision for ideas applicable to NLP with limited data. It highlights meta-learning and adversarial methods as ways to exploit auxiliary tasks or reduce source-specific representations.
- Meta-Learning: Meta-learning trains a model to use auxiliary high-resource tasks beneficially for a low-resource target task.The approach is based on multi-task learning and has been evaluated on sentiment analysis, user intent classification, and natural language understanding.
- Adversarial Methods: Adversarial discriminators can prevent transfer models from learning representations specific to a data source when pretraining and target-domain features differ.This addresses a transfer-learning issue in which neural models may incorporate uncontrolled source-specific information.
7 Discussion and Conclusion
The survey identifies cross-approach comparison as a major future need and emphasizes analyzing low-resource scenarios across multiple data-availability dimensions.
- Analyzing data availability across different dimensions can reveal which techniques apply to a specific low-resource setting.
- Fair comparisons across low-resource approaches are needed to guide practitioners in selecting suitable techniques.Such comparisons are difficult because methods require different kinds of auxiliary data.
- Future research should examine how approaches compare, where they complement one another, and how to combine them effectively.
A Existing Surveys on Low-Resource Topics and Languages
The paper situates its survey alongside a growing body of task- and language-specific reviews, which provide starting points for more focused reading.
- Task- and language-specific surveys can serve as starting points for more in-depth reading.
B Complexity of Tasks
The paper illustrates task complexity and resource variation across six languages, showing that higher-level applications and adequate data are unevenly available.
- Task coverage across languages: The task inventory spans basic applications such as tokenization and higher-level applications such as question answering, but is explicitly non-exhaustive.
- Resource coverage: Available resources include treebanks, named-entity-recognition corpora, multilingual embeddings, pretrained models, and benchmarks, with coverage varying by language and task.