PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Data Augmentation Approaches for NLP

Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, Eduard Hovy

arXiv:2105.03075v5cs.CLcs.AIcs.LG

TL;DR

NLP data augmentation is growing but remains underexplored because discrete language complicates augmentation and invariance. This paper presents a structured survey of representative methods and applications, concluding that substantial exploration remains and that the field lacks unified frameworks and benchmarks.

  • Problem

    NLP data augmentation remains relatively underexplored despite growing demand, partly because discrete language makes augmentation and invariance difficult.

  • Method

    The paper provides a comprehensive, structured survey of representative NLP augmentation techniques, applications, tasks, challenges, and future directions.

  • Results

    The survey organizes the literature and identifies substantial room for further exploration in NLP data augmentation.

  • Takeaways & Limitations

    The paper serves as a guide for choosing NLP augmentation techniques and motivates additional research in the area.

  • Takeaways & Limitations

    NLP augmentation lacks generalized techniques and standard unified benchmark tasks and datasets for comparing methods.

Abstract

from arXiv · show

Data augmentation has recently seen increased interest in NLP due to more work in low-resource domains, new tasks, and the popularity of large-scale neural networks that require large amounts of training data. Despite this recent upsurge, this area is still relatively underexplored, perhaps due to the challenges posed by the discrete nature of language data. In this paper, we present a comprehensive and unifying survey of data augmentation for NLP by summarizing the literature in a structured manner. We first introduce and motivate data augmentation for NLP, and then discuss major methodologically representative approaches. Next, we highlight techniques that are used for popular NLP applications and tasks. We conclude by outlining current challenges and directions for future research. Overall, our paper aims to clarify the landscape of existing literature in data augmentation for NLP and motivate additional work in this area. We also present a GitHub repository with a paper list that will be continuously updated at https://github.com/styfeng/DataAug4NLP

1 Introduction

Data augmentation increases training-example diversity without collecting new data, but its NLP adoption has lagged because discrete language makes invariance harder to preserve. This survey organizes the growing literature and identifies research gaps across methods, applications, and tasks.

  • Data augmentation increases training-example diversity without explicitly collecting new data.
  • NLP data augmentation remains comparatively underexplored because discrete language rules out continuous noising and complicates maintaining invariance.
  • Low-resource domains, expanding tasks, and large pretrained models create increasing demand for NLP data augmentation.
  • Research is especially sparse for nonclassification tasks such as span-based tasks and generation despite their real-world ubiquity.
  • The paper provides a detailed, structured survey intended to give a broad view of NLP data augmentation and identify key challenges.
  • The survey covers representative methods, applications, common NLP tasks, and future directions, with a continuously updated paper repository.

2 Background

NLP data augmentation modifies existing examples or generates synthetic data to increase diversity, often serving as a regularizer. Its methods trade implementation simplicity against variation, performance, and distributional fit, while theoretical understanding remains limited.

  • Data augmentation increases training-data diversity by modifying existing examples or creating synthetic data, aiming to reduce overfitting.
  • Ideal augmentation is easy to implement and improves model performance, but practical methods generally trade simplicity against effectiveness.
  • Rule-based methods are easier to implement but usually yield incremental gains, whereas model-based methods can provide stronger variation and performance at greater cost.
  • Augmented data should be neither too similar nor too different from the original, because either extreme can increase overfitting or reduce domain representativeness.
  • A KL-divergence-based procedure selects among augmentation heuristics in advance instead of comparing all heuristics, reducing a potentially time- and cost-intensive process.
  • The literature has limited theoretical explanation of why augmentation works, with regularization alone considered insufficient.

3 Techniques & Methods

The survey groups NLP augmentation into extensible primitives, interpolation-based methods, and model-based approaches. These range from simple token or structural transformations to embedding mixing, backtranslation, contextual replacement, and learned generation.

  • Representative NLP augmentation techniques are organized into rule-based, example interpolation-based, and model-based categories.
  • Rule-Based Techniques: Rule-based primitives apply predetermined transforms without model components, while feature-space methods augment representations rather than input data.
  • Rule-Based Techniques: EDA applies random token insertion, deletion, and swapping, while UDA uses consistency training on original and augmented example pairs.
  • Rule-Based Techniques: Dependency tree morphing swaps or deletes children sharing a parent, adapting structural transformations for dependency-annotated sentences.
  • Example Interpolation Techniques: MIXUP interpolates inputs and labels from multiple examples, with NLP variants mixing embeddings or higher hidden layers to avoid requiring continuous inputs.
  • Example Interpolation Techniques: SEQ2MIXUP mixes sequence pairs using either token-level binary masks or soft interpolation, with the soft version outperforming the hard version and earlier interpolation methods.
  • Model-Based Techniques: Model-based methods include backtranslation, contextual word replacement, pretrained-language-model generation, and corrupt-and-reconstruct augmentation.
  • Model-Based Techniques: Some approaches learn label-conditioned generators or select generated candidates using classifiers, while others use controlled paraphrasing and automated augmentation.

4 Applications

NLP data augmentation supports low-resource languages, bias mitigation, class balancing, few-shot learning, adversarial-example construction, and multiple task settings. The surveyed applications combine linguistic transformations, synthetic generation, and representation or sampling strategies.

  • Low-Resource Languages: Low-resource NLP applications use backtranslation, self-learning, rare-word contexts, and multilingual code-switching to expand training data.
  • Low-Resource Languages: Multilingual code-switching augmentation improved performance across 5 tasks spanning 19 languages.
  • Bias Mitigation: Gender swapping, counterfactual interventions, and predicate-argument augmentation are used to mitigate or assess bias robustness.
  • Class Imbalance: Class imbalance is commonly addressed through undersampling and oversampling, including SMOTE, MLSMOTE, and potentially EDA.
  • Few-Shot Learning: Few-shot augmentation generates examples for novel classes using learned analogy transformations, delta-network formulations, or language-model generation.
  • Few-Shot Learning: Theoretical few-shot formulations discussed here were applied only to image tasks, although their formulations are described as generally applicable.
  • Adversarial Examples: Adversarial examples are constructed through label-preserving transformations such as paraphrasing, word swapping, distractor spans, and WordNet relations.

5 Tasks

Data augmentation has been applied across diverse NLP tasks, with methods ranging from synthetic data generation and perturbation to task-specific augmentation frameworks. Reported applications include summarization, question answering, sequence tagging, parsing, grammatical error correction, machine translation, data-to-text generation, and dialogue.

  • Summarization: Backtranslation and consistency-based augmentation have been used for few-shot abstractive summarization, while synthetic and real data have supported iterative augmentation.WIKIREF provides an additional query-focused summarization dataset for augmentation.
  • Question Answering: Backtranslation, distant supervision, question generation, and cross-lingual substitution have been explored for domain-agnostic, open-domain, zero-shot, and multilingual question answering.QANet shows improved SQuAD performance when combined with backtranslation-generated augmented data.
  • Sequence Tagging Tasks: Token and synonym replacement improve named entity recognition across recurrent and transformer models, while SEQMIX improves NER and Event Detection for active sequence labeling.Other sequence-tagging methods include tag-and-word language-model generation and dependency-tree morphing for POS tagging.
  • Parsing Tasks: Parsing augmentation includes recombinant examples sampled from induced synchronous context-free grammars, compositional fragment replacement, dependency-tree morphing, and nonce sentences with controlled linguistic properties.These methods target semantic parsing and low-resource dependency parsing.
  • Grammatical Error Correction: Grammatical error correction augmentation uses external revision data, multi-task transfer, synthetic error injection, confusion sets, learned error patterns, and latent-representation or sequence-transduction noising.These approaches address the typical lack of parallel data for GEC.
  • Data-to-Text NLG: Data-to-text augmentation perturbs structured inputs, uses Wikipedia targets with parsed triples, permutes meaning-representation sequences, and samples decoder-hidden-state noise; these methods improve WebNLG or E2E-NLG performance.Game-summary generation can perturb score values while preserving the overall game outcome.

6 Challenges & Future Directions

The paper identifies unresolved theoretical, methodological, and domain-specific challenges for NLP data augmentation, while proposing directions including online augmentation, unified benchmarks, better reporting, and vision-inspired methods.

  • Challenges: Data augmentation research still lacks a clear theoretical account of why techniques work and how to measure their quality without full-scale experiments.Affinity and diversity may predict performance in vision, but their applicability to NLP remains unclear.
  • Challenges: Simple augmentation methods may provide little benefit for BERT and RoBERTa on well-represented, in-domain English classification tasks.The paper suggests studying whether augmentation is more effective for scarce or out-of-domain tasks.
  • Challenges: NLP augmentation remains difficult for multimodal, span-based, specialized-domain, and low-resource-language settings because modalities, dependencies, vocabulary, and linguistic differences constrain suitable transformations.Span-based methods must preserve dependencies across text locations, while specialized and low-resource settings may lack usable pretrained models, external resources, or high-resource cognates.
  • Future Directions: Future work could extend vision-inspired augmentation concepts to text and investigate simultaneous manipulation of multiple modalities.Examples include text analogues of grayscaling, morphing, and perspective changes, as well as negative data augmentation for text.
  • Future Directions: Online augmentation is a promising direction, but NLP methods often depend on external resources and task-specific invariances, so deciding when stochastic augmentation helps remains difficult.Consequently, offline generation and storage are common practices in NLP.
  • Future Directions: The field would benefit from generalized augmentation techniques, unified benchmark tasks and datasets, and practices that improve accessibility, reproducibility, transparency, and interpretability.Suggested practices include releasing code and augmented datasets, reporting variation, standardizing evaluation, analyzing hyperparameters, and documenting failures and theoretical intuition.

7 Conclusion

The paper presents a comprehensive, structured survey of NLP data augmentation, covering its foundations, representative techniques, applications, challenges, and future directions. It positions the survey as a guide for method selection and a stimulus for further research.

  • The survey explains data augmentation for NLP, including its background and how it works.
  • It reviews representative techniques and augmentation methods for popular NLP applications and tasks.
  • The paper outlines challenges and future directions, aiming to guide researchers and inspire additional work in NLP data augmentation.

A Useful Blog Posts and Code Repositories

The paper lists external blog posts and code repositories that supplement its discussion of text augmentation techniques and related tools.

  • The resources include introductory and detailed blog posts on text augmentation techniques.
  • The list includes nlpaug, a lightweight library for augmentation on text and audio.
  • The section provides supplementary links beyond the papers and methods discussed in the main text.

B DA Methods Table - Description of Columns and Attributes

Table 1 compares data augmentation methods by applicability, dependencies, preprocessing, modification depth, and task scope. These attributes capture required resources, where transformations occur, and how broadly methods may transfer across NLP tasks.

  • Dependencies and requirements: External knowledge records whether augmentation requires resources beyond the input examples and task definition, such as WordNet or PPDB.
  • Dependencies and requirements: The pretrained attribute indicates whether augmentation requires a pretrained model such as BERT or GPT-2.
  • Dependencies and requirements: Preprocessing specifies required steps such as tokenization or dependency parsing; a hyphen means none is required or the requirement is unstated.
  • Transformation scope: Level describes whether a method modifies only the input, both input and label, embedding or hidden representations, or higher task-model layers.
  • Applicability: Task-agnosticness estimates how readily a method extends across tasks, from broad applicability to restriction to one task or a small related group.

C Additional DA Works by Task

This section catalogs additional data-augmentation works across grammatical error correction, neural machine translation, dialogue, and multimodal tasks. Each listed work is accompanied by a brief description.

  • The section directs readers to Tables 2–5 for task-specific works, each described briefly.
  • Grammatical Error Correction: Additional data-augmentation works for grammatical error correction are collected in Table 2.
  • Neural Machine Translation: Additional data-augmentation works for neural machine translation are collected in Table 3.
  • Dialogue: Additional data-augmentation works for dialogue are collected in Table 4.
  • Multimodal Tasks: Additional data-augmentation works for multimodal tasks are collected in Table 5.
Loading 2105.03075v5…