PaperlayerView exact revision on arXiv

Source-linked AI summary

An Empirical Survey of Data Augmentation for Limited Data Learning in NLP

Jiaao Chen, Derek Tam, Colin Raffel, Mohit Bansal, Diyi Yang

arXiv:2106.07499v1cs.CLcs.AI

TL;DR

NLP needs methods that learn from limited labeled data because large datasets are costly to collect and update. This paper surveys and experimentally compares data augmentation across 11 NLP tasks, finding task- and learning-setting-dependent patterns rather than one universally best method. It also identifies assumptions about label and distribution preservation and limited transfer across datasets as important challenges.

  • Problem

    Large labeled datasets are expensive to collect and update, while no systematic empirical overview had established which NLP augmentations work in limited-data settings.

  • Method

    The paper categorizes recent NLP augmentation methods and compares them in supervised and semi-supervised experiments across 11 tasks.

  • Results

    No single augmentation works best universally: token-level methods generally perform well in supervised learning, while round-trip translation usually works best in semi-supervised learning.

  • Takeaways & Limitations

    Choosing augmentation should depend on the task and learning setting because augmentation can sometimes reduce performance.

  • Takeaways & Limitations

    Current methods often assume augmentations preserve labels and data distributions, but violations can introduce noisy labels, distribution shifts, and lower performance or generalization.

Abstract

from arXiv · show

NLP has achieved great progress in the past decade through the use of neural models and large labeled datasets. The dependence on abundant data prevents NLP models from being applied to low-resource settings or novel tasks where significant time, money, or expertise is required to label massive amounts of textual data. Recently, data augmentation methods have been explored as a means of improving data efficiency in NLP. To date, there has been no systematic empirical overview of data augmentation for NLP in the limited labeled data setting, making it difficult to understand which methods work in which settings. In this paper, we provide an empirical survey of recent progress on data augmentation for NLP in the limited labeled data setting, summarizing the landscape of methods (including token-level augmentations, sentence-level augmentations, adversarial augmentations, and hidden-space augmentations) and carrying out experiments on 11 datasets covering topics/news classification, inference tasks, paraphrasing tasks, and single-sentence tasks. Based on the results, we draw several conclusions to help practitioners choose appropriate augmentations in different settings and discuss the current challenges and future directions for limited data learning in NLP.

1 Introduction

Limited labeled data constrains NLP applications because collecting and updating high-quality annotations is expensive. The survey reviews augmentation methods and evaluates them across 11 NLP tasks to clarify which approaches work in different settings.

  • Large labeled datasets enable strong deep-learning performance but are expensive to collect and difficult to update as applications change.This motivates learning algorithms that can train with limited labeled data.
  • Data augmentation generates new examples by transforming existing data using prior knowledge about a task’s structure.Augmented data can support supervised or semi-supervised learning.
  • The survey reviews and categorizes recent NLP augmentation methods, compares them experimentally in supervised and semi-supervised settings, and discusses future directions.The experiments cover 11 NLP tasks.
  • No single augmentation works best for every task, but token-level methods tend to work well in supervised learning and sentence-level methods usually work best in semi-supervised learning.Augmentation can sometimes hurt performance, including in semi-supervised settings.
  • Unlike broader or task-specific surveys, this work focuses specifically on data augmentation and provides a more comprehensive, fine-grained review of recent methods.

2 Data Augmentation for NLP

NLP data augmentation operates at token, sentence, adversarial, and hidden-representation levels. These methods trade off semantic preservation, diversity, generalization, computational effort, and robustness.

  • Data augmentation increases dataset size and diversity by manipulating existing examples, helping address overfitting from limited labeled data.
  • Token-Level Augmentation: Token-level augmentation modifies words or phrases while aiming to retain the original sentence’s meaning and labels.
  • Token-Level Augmentation: Synonym replacement uses dictionaries or embedding similarity, but its improvements are usually minimal and can be unreliable.
  • Token-Level Augmentation: Random insertion, replacement, deletion, and swapping are efficient noise-injection methods that can improve simple models trained on small datasets.Their gains may be unstable when perturbations change sentence meaning, and fine-tuning large pretrained models can attenuate improvements.
  • Token-Level Augmentation: Compositional augmentation recombines fragments from different sentences using carefully designed rules to target compositional and out-of-distribution generalization.It has been applied to sequence labeling and text generation.
  • Sentence-Level Augmentation: Sentence-level augmentation modifies entire sentences; paraphrasing creates diverse text with different wording and structure while preserving meaning.Round-trip translation translates through an intermediate language and back to generate paraphrases.
  • Sentence-Level Augmentation: Conditional generation produces new text from a label-conditioned language model and often uses filtering, but requires significant training effort.
  • Adversarial Data Augmentation: Adversarial augmentation adds perturbations that strongly influence predictions and confidence without changing human judgments, supporting robustness and generalization.White-box methods use accessible model parameters or gradients, while black-box methods rely on model-agnostic task heuristics.

3 Consistency Training with DA

Data augmentation supports semi-supervised learning by enforcing consistent predictions between clean and perturbed inputs. The surveyed approach uses consistency losses and reports that consistency training can be effective for NLP.

  • Consistency-training SSL combines labeled and unlabeled examples and uses augmentation to regularize predictions under input perturbations.
  • The consistency objective compares a fixed model’s prediction on a clean unlabeled input with the trainable model’s prediction on its augmented version.The clean prediction acts as a synthetic target distribution for the augmented input.
  • Table 2 reports topic and news classification results using 10 examples, averaged over three random seeds with 95% confidence intervals.For PubMed, both accuracy and F1 score are reported.
  • The consistency term can use measures such as cross entropy, KL divergence, or mean-squared error, with λu controlling its weight.
  • Consistency training has been shown to work effectively for semi-supervised NLP, alongside confidence thresholding, training signal annealing, and entropy minimization.

4 Empirical Experiments

The experiments compare ten model-agnostic augmentation techniques across 11 NLP tasks using extremely limited labeled data. Results vary by task and learning setting: token-level methods often lead supervised learning, while sentence-level methods are strongest for semi-supervised learning.

  • Experimental design: The study compares ten augmentation techniques across news/topic classification, inference, similarity/paraphrase, and single-sentence tasks.The experiments use benchmark datasets spanning these task families, with results reported for limited-data supervised and semi-supervised settings.
  • Experimental design: The experiments use 10 labeled examples per class, with an additional 5000 unlabeled examples per class in the semi-supervised setup.BERT-base is used with common hyperparameters across datasets and methods.
  • News/Topic Classification Tasks: Token-level augmentations work best in supervised news and topic classification, while round-trip translation usually ranks highest in semi-supervised settings.Word replacement performs among the best in supervised settings, and round-trip translation reaches the highest or second-highest score consistently in semi-supervised settings.
  • Inference Tasks: Token-level augmentations perform best overall for both supervised and semi-supervised inference tasks, despite inference labels depending heavily on particular words.The reported methods include random insertion, random deletion, and word replacement.
  • Similarity and Paraphrase Tasks: Random swapping leads supervised similarity and paraphrase tasks, while hidden-space cutoff performs best on average in semi-supervised settings.The paper notes that text paraphrasing can change whether two texts are paraphrases, making augmentation choices consequential.
  • Single Sentence Tasks: Hidden-space cutoff gives the largest supervised boost on single-sentence tasks, whereas round-trip translation works best semi-supervised; most methods hurt CoLA performance.The authors suggest this may reflect a mismatch between preserving meaning and preserving grammatical correctness.
  • Overall results: No single augmentation is best for every task, and augmentation can sometimes reduce performance even in semi-supervised learning.Token-level methods are generally effective for supervised learning, while round-trip translation gives the most consistent semi-supervised gains; cutoff may suit limited computation.

5 Other Limited Data Learning Methods

The paper situates data augmentation alongside other approaches for learning with limited labeled data in NLP. These include cross-lingual transfer, semi-supervised learning methods using unlabeled data, and few-shot learning using prior knowledge.

  • Low-Resourced Languages: Cross-lingual transfer learning addresses low-resource languages by transferring models from resource-rich to resource-poor languages.Low-resource languages often lack large monolingual or parallel corpora and sufficient manually crafted linguistic resources.
  • Other Methods for Semi-Supervised Learning: Semi-supervised learning reduces reliance on labeled data by incorporating large amounts of unlabeled data, which is usually cheaper and easier to obtain.This paper focuses on consistency regularization, while other NLP approaches include self-training, generative methods, and co-training.
  • Few-shot Learning: Few-shot learning uses prior knowledge, such as pretrained models or supervised data from other domains and modalities, rather than relying only on unlabeled data.The paper notes growing adoption of few-shot learning in NLP.

6 Discussion and Future Directions

The discussion identifies unresolved challenges in ensuring that augmentations preserve labels and distributions and in selecting methods that generalize across tasks and datasets. It points toward theoretical guarantees and automated augmentation-policy search.

  • Scope: The work compares data augmentation methods on 11 NLP tasks and uses the discussion to identify challenges and future research directions.The stated focus is limited-data learning in NLP, especially data augmentation and consistency regularization.
  • Theoretical Guarantees and Data Distribution Shift: Augmentation assumptions about label preservation and unchanged data distributions often fail, producing noisy labels or distribution shifts that can reduce performance or generalization.The paper cites QQP as an example and calls for guarantees under specified conditions.
  • Automatic Data Augmentation: Current augmentation methods are generally manually designed, and automatic selection of appropriate augmentations remains under-investigated.The authors propose learning augmentation strategies or searching for task-, dataset-, and model-specific policies.
  • Automatic Data Augmentation: Augmentations effective for one dataset or task often fail to transfer, as paraphrasing may change labels in subtle settings such as bias classification.The paper presents automated policy search as a possible route to better generalizability.

A Experimental Setup

The experimental setup uses NVIDIA 2080ti and V-100 GPUs, BERT-base, shared hyperparameters, and validation-based checkpoint selection across datasets.

  • Compute and model: Supervised experiments take 20 minutes, while semi-supervised experiments take two hours on NVIDIA 2080ti and V-100 GPUs.The BERT-base model has 100M parameters.
  • Hyperparameters: The experiments use the same hyperparameters across datasets, selecting the best model checkpoint only through validation performance.The stated settings include learning rate 2e−5, batch size 16, unlabeled-to-labeled ratio 3, and dropout ratio 0.1.

B Results for 100 Labeled Data per Class

Results across task families show that augmentation effectiveness depends strongly on the task and learning setting: no method is uniformly best. Token-level methods work well for supervised learning, sentence-level methods usually perform best semi-supervised learning, and augmentation can reduce performance.

  • News/Topic Classification Tasks: Topic and news classification methods perform similarly, within 2 points of one another, in supervised and semi-supervised settings.The reported results indicate that augmentation is useful with limited labeled data, while its effectiveness diminishes as more labeled data becomes available.
  • Inference Tasks: Most augmentation methods hurt inference-task performance in both settings, with a greater drop in semi-supervised learning.
  • Similarity and Paraphrase Tasks: Most augmentation methods hurt similarity and paraphrase-task performance in both settings, especially semi-supervised learning.
  • Single Sentence Tasks: Single-sentence augmentations generally improve performance except on CoLA, where hidden-level methods work well and Cutoff performs best.The paper hypothesizes that most augmentations preserve meaning rather than grammatical correctness, which CoLA measures.

C Case Study

The case study examines whether augmentation preserves labels by inspecting examples from 20 Newsgroups and RTE. Label preservation is task-dependent: no flips occurred for 20 Newsgroups, whereas RTE showed substantially different flip rates across methods.

  • The case study evaluates label preservation using 25 examples from the best- and worst-performing methods on 20 Newsgroups and RTE.
  • 20 Newsgroups: For 20 Newsgroups, Random Deletion performed best and Language Model worst, with no observed label flips for either method.The authors relate this to inputs usually containing several paragraphs and multiple references to the topic.
  • RTE: For RTE, Language Model was worst and Cutoff best; Language Model flipped 24% of labels, while Cutoff flipped 4%.Language Model also had 4% uncertain examples, compared with 12% for Cutoff.
Loading 2106.07499v1…