PaperlayerView exact revision on arXiv

Source-linked AI summary

Data Augmentation Approaches in Natural Language Processing: A Survey

Bohan Li, Yutai Hou, Wanxiang Che

arXiv:2110.01852v3cs.CLcs.AIcs.LG

TL;DR

Existing NLP data-augmentation surveys provide limited or overly broad method categories, motivating a more inclusive account organized around augmented-data diversity. This survey analyzes paraphrasing, noising, and sampling methods, their NLP applications, and current challenges, finding distinct patterns in learnability, diversity, fluency, and task use.

  • Problem

    Existing NLP data-augmentation surveys use categories that can be too limited or general, motivating an inclusive overview of methods.

  • Method

    The survey categorizes data-augmentation methods by augmented-data diversity into paraphrasing, noising, and sampling, then analyzes their characteristics, applications, strategies, and challenges.

  • Results

    Sampling-based methods show clear momentum in text classification and text generation, while paraphrasing- and noising-based methods remain common in structured prediction.

  • Takeaways & Limitations

    The three diversity-based categories help researchers understand and develop data-augmentation methods and identify strategies for improving model performance.

  • Takeaways & Limitations

    NLP data augmentation still lacks systematic probing and theoretical analysis explaining why methods work.

Abstract

from arXiv · show

As an effective strategy, data augmentation (DA) alleviates data scarcity scenarios where deep learning techniques may fail. It is widely applied in computer vision then introduced to natural language processing and achieves improvements in many tasks. One of the main focuses of the DA methods is to improve the diversity of training data, thereby helping the model to better generalize to unseen testing data. In this survey, we frame DA methods into three categories based on the diversity of augmented data, including paraphrasing, noising, and sampling. Our paper sets out to analyze DA methods in detail according to the above categories. Further, we also introduce their applications in NLP tasks as well as the challenges. Some helpful resources are provided in the appendix.

1. Introduction

This survey organizes NLP data augmentation around the diversity of augmented data, covering paraphrasing, noising, and sampling while reviewing applications, strategies, related topics, and challenges.

  • Survey scope: The survey frames NLP data augmentation into paraphrasing, noising, and sampling according to augmented-data diversity.It presents this organization as a way to provide an inclusive overview and examine why data augmentation works.
  • Taxonomy: Paraphrasing makes limited changes, noising introduces more continuous or discrete changes, and sampling generates new data from the original distribution.Sampling can produce brand new data using artificial heuristics and trained models.
  • Survey scope: The paper analyzes each category’s methods and characteristics, including granularity and level.The review covers every method in the three categories.
  • Strategies: It summarizes strategies for improving augmented-data quality, including method stacking, optimization, and filtering.
  • Applications and connections: The survey examines applications across NLP tasks, presents their development through a timeline, and connects data augmentation with related topics.Related topics include pretrained language models, contrastive learning, generative adversarial networks, and adversarial attacks.
  • Challenges: The paper identifies theoretical narrative and generalized methods as challenges and future development directions.

2. Data Augmentation Methods in NLP

The survey defines augmentation as producing valid additional training data while seeking greater diversity, and classifies methods as paraphrasing, noising, or sampling.

  • Validity and diversity: Augmented data must remain valid for the task and belong to the same distribution as the original data.Examples include preserving similar semantics in machine translation and the same label in text classification.
  • Taxonomy: The survey categorizes methods by augmented-data diversity into paraphrasing, noising, and sampling.This classification is presented as distinct from categorizing methods by implementation technique.
  • Paraphrasing: Paraphrasing uses restrained changes to produce data conveying information very similar to the original.
  • Noising: Noising adds discrete or continuous noise while maintaining validity, with robustness as its stated purpose.
  • Sampling: Sampling models data distributions to generate novel, more diverse data using artificial heuristics and trained models.
  • Paraphrasing: The survey presents paraphrasing methods across word-level, phrase-level, and sentence-level granularity.
  • Taxonomy: Across the three categories, paraphrasing, noising, and sampling provide progressively greater diversity than the original input.

Paraphrasing-based Methods

Paraphrasing-based augmentation generates alternative expressions intended to preserve the original information while increasing training-data diversity. The survey organizes methods by lexical, phrase, and sentence-level transformations, including thesauruses, embeddings, language models, rules, and translation.

  • Paraphrases provide alternative ways to convey the same information and can therefore serve as augmented data.
  • Thesauruses: Thesaurus methods replace words with synonyms or hypernyms, but substitution range, part of speech, ambiguity, and excessive semantic changes limit them.WordNet is commonly used as an external resource, and more similar synonyms may be selected with greater probability.
  • Semantic Embeddings: Semantic-embedding methods replace words with nearby words in embedding space, offering broader replacement coverage while retaining ambiguity problems.Word2Vec, GloVe, and FastText are examples of pretrained embeddings used for nearest-neighbor replacement.
  • Language Models: Language-model methods predict masked words from context, which alleviates ambiguity and incorporates contextual semantics but remains limited at the word level.Masked language models such as BERT and RoBERTa can fill masked positions, while conditional models can incorporate annotated labels.
  • Rules: Rule-based methods use dictionaries, regular expressions, or dependency transformations to generate paraphrases, preserving semantics but requiring heuristics and offering limited variation.Examples include expanding or abbreviating expressions and changing sentence voice through dependency-tree transformations.
  • Machine Translation: Machine translation and Seq2Seq generation produce paraphrases through translation systems or directly trained generative models.Translation is described as a popular augmentation approach, while Seq2Seq models can generate more diverse sentences given suitable training objectives.

2.1.6. Model Generation

Model-generation methods use trained generative architectures to create augmented examples, while noising methods deliberately perturb original text. The section also describes word, sentence, and instance-level noise operations and their label-preservation risks.

  • 2.1.6. Model Generation: Seq2Seq, Transformer, GAN, and pretrained-model approaches generate new utterances, reconstructed fragments, or questions for different NLP tasks.Generation can use delexicalized utterances, diverse-rank inputs, masked sentences with labels, or prior information from pretrained models.
  • 2.1.6. Model Generation: Model-generation methods have wide application potential but require training data and involve high training difficulty.
  • 1. Noising-based methods: Noising-based methods add weak continuous or discrete noise so augmented examples deviate from originals without seriously affecting semantics.The survey contrasts this with paraphrasing, whose goal is to keep augmented semantics as similar to the original as possible.
  • Noising methods: Word and sentence swapping can perturb order while retaining readable or task-relevant information, including labels in some task-specific procedures.Methods range from swapping randomly selected words to combining same-label tweet segments or shuffling labeled token segments.
  • Noising methods: Deletion removes randomly selected words or sentences, with applications including dialogue-act combinations and legal-document augmentation.Attention mechanisms can determine deletion targets, and sentence deletion may exploit irrelevant statements in legal documents.
  • Noising methods: Insertion and substitution add or replace text, but noisy information can change labels unless resources or samples are chosen to preserve label relevance.Recommended safeguards include label-independent external resources for word-level operations and same-label samples for sentence-level operations.

1. Noising-based methods

The survey identifies poor interpretability and limited diversity for individual noising-based methods as disadvantages.

  • Noising-based methods can improve model robustness but have poor interpretability and limited diversity for each individual method.

Sampling-based Methods

Sampling-based methods learn or approximate the original data distribution to generate new, task-specific samples, often increasing diversity beyond paraphrasing and noising. Their greater diversity and fluency come with greater complexity, training requirements, and limited document-level applicability.

  • Definition: Sampling-based methods generate new data from the original distribution and require task information such as labels and data format.They use rules or trained models to produce task-specific samples while ensuring validity and increasing diversity.
  • Rule-based methods: Rule-based sampling directly constructs valid samples with heuristics, but the outputs are not guaranteed to resemble the originals or preserve labels.Rules may determine labels, as in subject-object swaps and passive-voice transformations.
  • Learnable methods: Learnable sampling methods include non-pretrained Seq2Seq models and pretrained language models that generate synthetic data from learned distributions.Non-pretrained models offer strong diversity and application breadth but require training data and are difficult to train; pretrained models also require training data.
  • Comparative characteristics: Sampling-based methods are generally more diverse and fluent than paraphrasing- and noising-based methods because they are usually learnable and more complex.Rule-based sampling is an exception to the broader learnability pattern.
  • Comparative characteristics: Mixup is the only online learnable method, generating cross-label discrete embeddings during model training.Unlike offline methods, its augmentation process occurs within model training.
  • Comparative characteristics: Most non-learnable methods require external knowledge, whereas sampling-based methods are task-related and construct text and labels for specific tasks.Sampling-based augmentation therefore incorporates task definitions during data construction.
  • Limitations: Learnable methods generally operate at sentence level, so document augmentation still relies on simpler non-learnable methods.The authors identify weaker document-processing ability as the reason learnable methods do not support document augmentation.

3. Strategies and Tricks

The survey presents combinations, optimization choices, training strategies, and filtering as ways to improve augmented-data quality and augmentation effects. These choices address variation, scale, hyperparameter sensitivity, and noise or errors introduced during augmentation.

  • Method stacking: Different augmentation methods can be combined to improve performance through richer, complementary, or multi-source augmented data.Combinations may use methods within the same category or combine different sources and techniques.
  • Method stacking: Unsupervised methods such as EDA and UDA are grouped and widely used when simple, task-independent augmentation meets the application demand.EDA combines synonym replacement, insertion, swapping, and deletion, while UDA combines back-translation with unsupervised noising.
  • Method stacking: Applying augmentation at multiple granularities enriches data with varied changes and can improve model robustness.Examples apply methods at word, phrase, sentence, embedding, or frame levels.
  • Optimization: Optimization choices include how augmented data is used, hyperparameters, training strategies, and training objects.These four angles organize the survey’s discussion of augmentation optimization.
  • Optimization: Augmented data may be used for pretraining or direct training depending on quality, while excessive augmentation may require oversampling the original data or other balancing practices.The survey distinguishes data-use decisions by both quality and relative amount.
  • Filtering: Filtering is used before or after augmentation because augmentation can introduce noise or errors.Input filtering removes unsuitable examples, while output filtering uses discriminators or classifiers to ensure synthetic-data quality.

4. Applications on NLP Tasks

Data augmentation is used most widely for text classification, while task characteristics shape method preferences: text generation favors sampling for semantic diversity, whereas structured prediction favors paraphrasing for format validity. The survey also reports growing adoption across all three task types.

  • Overview: Direct performance comparisons across augmentation methods are difficult because tasks, metrics, datasets, architectures, and experimental settings differ.The survey therefore analyzes methods separately across text classification, text generation, and structured prediction.
  • Text Classification: Text classification is the most common application because augmentation mainly needs to preserve the semantics relevant to predicting a closed-set label.The task maps input text directly to a category.
  • Task preferences: Text generation favors sampling-based methods for semantic diversity, whereas structured prediction favors paraphrasing-based methods because it is sensitive to output format.Structured prediction consequently has higher requirements for data validity.
  • Method adoption: Simple unsupervised methods remain popular, while learnable paraphrasing and pretrained sampling methods attract attention for their diversity and effectiveness.Examples include machine translation, thesaurus-based paraphrasing, random substitution, model generation, and pretrained models.
  • Development: DA applications have increased over time; sampling-based methods show momentum in text classification and generation, while paraphrasing and noising remain common for structured prediction.The timeline distinguishes paraphrasing, noising, and sampling across the three task types.

5. Related Topics

The survey situates data augmentation alongside pretrained language models, contrastive learning, other data-manipulation methods, GANs, and adversarial attacks. These connections clarify overlapping uses while distinguishing their objectives and mechanisms.

  • Pretrained language models: Pretrained language models implicitly introduce additional training data through self-supervised learning on large-scale unsupervised data.Their constructed auxiliary supervision teaches representations for downstream tasks.
  • Contrastive learning: Contrastive learning uses data augmentation to construct similar same-label samples before selecting negative samples and learning an embedding space.The augmentation step supports the positive-pair construction stage of contrastive learning.
  • Related data manipulation: Oversampling reuses minority-group examples, whereas data augmentation generates augmented data; data cleaning and weighting instead modify quality or training importance.These methods are related because they can improve generalization but differ in how they manipulate data.
  • Generative adversarial networks: GAN generators may be used for augmentation, but GAN training primarily pits a generative model against a discriminator rather than directly producing training data.The survey notes that GAN-based augmentation has demonstrated effectiveness in some settings.
  • Adversarial attacks: Some adversarial-attack work uses augmentation methods such as code-switch substitution to generate adversarial examples for consistency regularization.Adversarial attacks target model mistakes, distinguishing this use from ordinary synthetic-data augmentation.

6. Challenges and Opportunities

The survey identifies unresolved challenges in NLP data augmentation, including limited theory, uncertain use with pretrained models, insufficient diversity in few-shot settings, and gaps across tasks, languages, and text lengths.

  • Theoretical Narrative: NLP data augmentation lacks systematic probing and theoretical analysis explaining why methods work beyond downstream-task effectiveness.The discrete nature of natural language makes theoretical understanding especially important.
  • More Exploration on Pretrained Language Models: Most pretrained-language-model augmentation work remains limited to masked-token completion, direct generation after fine-tuning, or self-training.The survey calls for more diverse and higher-quality generation using pretrained-model information at lower cost.
  • Few-shot Scenarios: Traditional paraphrasing dominates few-shot augmentation, but its semantic diversity is insufficient and pretrained models already cover some of its information.The survey highlights generating high-quality augmented data as an open route to faster generalization in few-shot scenarios.
  • Retrieval Augmentation: Retrieval augmentation is presented as a related direction that improves parameter efficiency and enables domain adaptation by updating retrieval data.The survey notes that retrieval-augmented models can store less knowledge in parameters and adapt domains through retrieval-corpus updates.
  • More Generalized Methods for NLP: No single augmentation method is currently effective for all NLP tasks because language is discrete and NLP includes task types such as structured prediction.This creates a gap between general augmentation methods and the requirements of diverse NLP tasks.
  • Working with Long Texts and Low Resources Languages: Existing methods progress mainly on short texts and common languages, while long texts remain difficult and low-resource languages lack augmentation resources.Low-resource languages have greater demand for augmentation despite scarce methods.

7. Conclusion

The paper presents a structured survey of NLP data augmentation, organizing methods by augmented-data diversity and connecting them to applications, related topics, resources, and future directions.

  • 7. Conclusion: The survey provides a comprehensive and structured overview of data augmentation for natural language processing.Its stated purpose is to inspect the nature of NLP data augmentation.
  • 7. Conclusion: It frames methods into paraphrasing, noising, and sampling according to the diversity of augmented data.The categories are intended to help researchers understand and develop data augmentation methods.
  • 7. Conclusion: The survey covers method characteristics, NLP-task applications, a development timeline, practical strategies, related topics, challenges, and research opportunities.The related-topic discussion distinguishes data augmentation from neighboring areas.

Appendix A. Related Resources

The appendix collects data-augmentation resources, repositories, links, and surveys, including tools for English and several other languages.

  • Appendix A. Related Resources: The appendix provides popular resources offering information or APIs for NLP data augmentation.These resources are intended to support access to augmentation information and tooling.
  • Appendix A. Related Resources: The appendix also lists a visual survey of NLP data augmentation and a Chinese word-and-phrase embedding corpus.These entries complement the implementation repositories with survey and corpus resources.
  • Appendix A. Related Resources: The appendix includes TextAttack for generating adversarial examples for NLP models and AugLy as a library for audio, image, text, and video augmentation.These resources extend beyond ordinary text-augmentation repositories.
  • Appendix A. Related Resources: It includes repositories such as NL-Augmenter, EDA, Unsupervised Data Augmentation, nlpaug, TextAttack, and AugLy.The listed tools cover augmentation and adversarial-example generation for text, with AugLy spanning multiple media types.
  • Appendix A. Related Resources: Resources are identified for Turkish and Chinese, including nlpaug and multiple Chinese augmentation repositories.The language-specific entries include EDA, BERT, BART, contextual augmentation, and PaddleNLP-related resources.
Loading 2110.01852v3…