Data Augmentation Approaches in Natural Language Processing: A Survey

1. Introduction

• Data augmentation increases data with modified or synthetic examples, alleviating data scarcity but remaining more difficult and underexplored for discrete natural language than for images.The survey notes that augmentation was widely used in computer vision before being introduced to NLP.Source: §1 · p. 4
• The survey categorizes NLP data augmentation by augmented-data diversity into paraphrasing, noising, and sampling to explain why augmentation works.Paraphrasing makes limited changes, noising introduces more continuous or discrete changes, and sampling generates the most diverse new data from the original distribution.Source: §1 · p. 4
• The paper reviews the three categories and their method characteristics, including granularity and level, then summarizes method stacking, optimization, and filtering strategies.These discussions are presented in Sections 2 and 3.Source: §1 · p. 4
• The survey covers applications of augmentation methods across NLP tasks, traces their development through a timeline, and identifies challenges including theoretical narrative and generalized methods.It also connects augmentation with related topics such as pre-trained language models, contrastive learning, generative adversarial networks, and adversarial attacks.Source: §1 · pp. 4–5

• Data augmentation generates additional synthetic training data for insufficient-data settings, using rule-based or learnable methods while preserving task validity and the original data distribution [27].Validity means preserving similar semantics in machine translation or the same label in text classification.Source: §2 · p. 6
• The survey organizes augmentation methods by augmented-data diversity into paraphrasing, noising, and sampling, which provide progressively broader diversity.The categorization is based on diversity while retaining validity as a prerequisite.Source: §2 · pp. 6–7
• Paraphrasing makes restrained word-, phrase-, or sentence-level changes that preserve very similar information and limited semantic difference from the original.The method produces augmented data conveying nearly the same information as the original form.Source: §2 · p. 6
• Noising adds discrete or continuous noise while guaranteeing validity, with the goal of improving model robustness.Source: §2 · p. 6
• Sampling-based methods learn data distributions and sample novel data within them, producing more diverse outputs for downstream needs through heuristics and trained models.Source: §2 · p. 6

Paraphrasing-based Methods

• Paraphrasing-based methods augment data by generating alternative expressions at lexical, phrase, and sentence levels while conveying the same information.The survey organizes these techniques according to the level at which paraphrases are generated.Source: Paraphrasing-based Methods · p. 7

2.1.1. Thesauruses

• Thesaurus methods replace selected words with synonyms or hypernyms, using resources such as WordNet and similarity-weighted sampling to preserve semantics.They are easy to use but have limited scope and part-of-speech coverage, cannot resolve ambiguity, and may alter semantics after excessive substitution.Source: §2.1.1 · pp. 7–8

2.1.2. Semantic Embeddings

• Semantic-embedding methods replace words with nearby embedding-space neighbors, providing higher replacement hit rates and broader coverage than thesauruses but retaining ambiguity problems.Word and frame embeddings were applied to Twitter messages, while other work combined embedding retrieval with thesaurus filtering.Source: §2.1.2 · pp. 8–9, Semantic Embeddings · p. 9

2.1.3. Language Models

• Language-model methods mask words and fill them contextually with BERT, RoBERTa, or related models, alleviating ambiguity while remaining largely word-level.Conditional BERT incorporates annotated labels, and other approaches combine BERT word-piece masking with GloVe neighbors to generate varied sentences.Source: §2.1.3 · p. 9, Language Models · p. 10

2.1.4. Rules

• Rule-based methods use dictionaries, regular expressions, or dependency-tree transformations to create word-, phrase-, and sentence-level paraphrases while preserving semantics.Their advantages are ease of use and semantic preservation, but they require artificial heuristics and offer limited coverage and variation.Source: §2.1.4 · p. 10

2.1.5. Machine Translation

• Machine translation provides a widely applicable paraphrasing method, with back-translation rewriting entire sentences rather than directly replacing individual words.Back-translation translates text into another language and then back, using trained models or translation APIs.Source: §2.1.5 · p. 11
• Back-translation has been used for English–French paraphrases, unsupervised augmentation, and style transfer, with variants adding temperature control, adversarial training, or discriminator filtering.These extensions target diversity, informative examples, or improved augmented-data quality.Source: §2.1.5 · p. 11
• Unidirectional translation generates multilingual or pseudo-parallel data by translating once, supporting cross-lingual embeddings and multiplied corpora with multilingual models.Its applications include constructing pseudo-parallel corpora for unsupervised cross-lingual word embeddings and translating English data into several languages.Source: §2.1.5 · pp. 11–12

2.1.6. Model Generation

• Model generation uses Seq2Seq, Transformer, GAN, and pretrained models to create diverse paraphrases, reconstructed fragments, or answerable and unanswerable questions across NLP tasks.Hou et al. generate utterances conditioned on a diverse rank, while Hou et al. use duplication-aware attention and diverse-oriented regularization; Li et al., Kober et al., and Liu et al. apply related architectures to other tasks.Source: §2.1.6 · pp. 12–13
• Model generation has wide application potential but requires training data and involves high training difficulty.The survey contrasts paraphrasing’s semantic similarity objective with noising methods that deliberately introduce faint deviations to expand data and improve robustness.Source: §2.1.6 · p. 13

2.2.1. Swapping

• Swapping randomly reorders words, labeled segments, sentences, or document halves within reasonable limits, exploiting humans’ tolerance for slight order changes.Wei et al. repeat word swaps proportionally to sentence length; Dai et al. preserve label order, while Luque et al. and Yan et al. apply larger-scale swapping.Source: §2.2.1 · p. 13

2.2.2. Deletion

• Deletion removes words or sentences randomly, including slot values in dialogue acts, while attention mechanisms can select deletion targets.Yan et al. delete potentially irrelevant legal-document sentences because case understanding should remain unaffected.Source: §2.2.2 · p. 14

2.2.3. Insertion

• Insertion adds synonym words or same-label sentences, and dialogue-act slot values, to create additional combinations.Yan et al. select sentences from other legal documents with the same label, while Wei et al. insert randomly positioned synonyms.Source: §2.2.3 · pp. 14–15
• Because random insertion may change the original label, word-level methods should use label-independent resources and sentence-level methods should use same-label samples.These safeguards are stated as practical tips for controlling newly introduced noisy information.Source: §2.2.3 · p. 15

2.2.4. Substitution

• Substitution replaces words or sentences with misspellings, placeholders, vocabulary items, same-label tokens, other-language words, slot values, or generated random strings.Examples include Coulombe et al. [7] and Regina et al. [10] using common misspellings, Qin et al. using multilingual code-switching, and task-specific slot substitution.Source: §2.2.4 · p. 15
• Random substitution can introduce noisy information that changes labels, so resources should be label-independent at word level or same-label at sentence level.Unlike paraphrasing, substitution generally avoids strings semantically similar to the original data.Source: §2.2.4 · pp. 15–16

1. Noising-based methods

• Noising-based methods improve model robustness, but suffer from poor interpretability and limited diversity for each method.Source: §1 · p. 16

Sampling-based Methods

• Sampling-based methods model the data distribution to generate task-specific samples using labels, data formats, artificial heuristics, or trained models, ensuring validity while increasing diversity.Their task-specific design makes them more flexible but more difficult than the other two categories.Source: Sampling-based Methods · p. 16

2.3.1. Rules

• Rule-based methods directly construct valid augmented data with linguistic or label heuristics, but they may differ from the original data and offer low coverage and limited variation.Examples include subject-object inversion, mathematical-equation reconstruction, logical inference, and rule-based positive or negative pair construction [25].Source: §2.3.1 · pp. 16–17

2.3.2. Non-pretrained Models

• Non-pretrained models commonly use back translation or task-specific Seq2Seq generation to learn mappings between distributions and produce highly diverse, broadly applicable data.They require training data and have high training difficulty [13, 26, 27, 74, 78, 79, 80].Source: §2.3.2 · pp. 17–18, Non-pretrained Models · p. 18

2.3.3. Pretrained Models

• Pretrained language models generate augmented data using learned linguistic knowledge, including filtered GPT-2 outputs, masked-language-model reconstruction, and autoregressive generation.These methods provide strong diversity and application but require training data [29] [21] [83] [44].Source: §2.3.3 · pp. 18–19, Pretrained Models · p. 19

2.3.4. Self-training

• Self-training converts unlabeled data into valid labeled samples with fine-tuned or iteratively updated teacher models, making it easier than generative methods and suitable for data-sparse scenarios.Its main requirement is access to unlabeled data.Source: §2.3.4 · pp. 19–20

2.3.5. Mixup

• Mixup samples virtual embeddings rather than natural-language text by interpolating word embeddings or sentence hidden states, enabling augmentation across different labels but reducing interpretability [35] [90] [91].Its variants include wordMixup, senMixup, Padv, Paut, Mixup-Transformer, IntraLADA, and InterLADA.Source: §2.3.5 · p. 20

2.4. Analysis

• Most sampling-based methods are learnable except rule-based methods, making them more complex and generally more diverse and fluent than paraphrasing- and noising-based methods.Among learnable methods, Mixup is the only online method and uniquely produces cross-label discrete embeddings.Source: §2.4 · p. 21
• Sampling-based methods are task-related and operate at text and label levels, while learnable methods usually provide sentence-level augmentation; document augmentation therefore still relies on non-learnable methods.Most non-learnable methods require external resources or heuristics, whereas learnable methods depend on training objectives or pretrained knowledge.Source: §2.4 · pp. 21–22

3. Strategies and Tricks

• The section frames these strategies as practical guidance for selecting and constructing suitable augmentation methods because their effects depend on application choices.The three augmentation types are paraphrasing, noising, and sampling.Source: §3 · p. 22

3.1. Method Stacking

• Combining augmentation methods can improve data richness, robustness, and performance through complementary same-type methods, unsupervised toolkits, and multi-granularity application.Examples include thesaurus-plus-embedding paraphrasing, EDA, UDA, and applying methods at word and sentence levels.Source: §3.1 · pp. 22–23

3.2. Optimization

• Optimization quality depends on how augmented data are used, hyperparameters, training strategies, and training objects.These four angles structure the section’s optimization guidance.Source: §3.2 · p. 23

3.2.1. The Use of Augmented Data

• Augmented data of low quality can pre-train models, whereas higher-quality data can train them directly; excessive amounts favor original-data oversampling or augmented-data pretraining followed by fine-tuning.The choice depends on augmented-data quality and its amount relative to the original data.Source: §3.2.1 · p. 23

3.2.2. Hyperparameters

• Hyperparameters substantially affect augmentation outcomes, with common parameters summarized in Figure 12.The figure organizes hyperparameters affecting each data augmentation method.Source: §3.2.2 · p. 23, Figure 12 · p. 24

3.2.3. Training Strategies

• Training strategies extend basic augmentation through adversarial training, optimization of generated outputs, reinforcement learning, and Generative Adversarial Networks.Qu et al., Quteineh et al., Hu et al., Liu et al., and works [62] [95] exemplify these extensions.Source: §3.2.3 · p. 23

3.2.4. Training Objects

• Training objects can be tuned to preserve semantic meaning while increasing diversity, generate diverse sentences, or emphasize difficult examples.Nugent et al. use softmax temperatures, Hou et al. use duplication-aware attention and diverse-oriented regularization, and Cheng et al. use curriculum learning.Source: §3.2.4 · p. 24

3.3. Filtering

• Filtering mitigates noise and errors introduced during augmentation by screening unsuitable inputs and low-quality synthetic outputs.Input filtering includes removing overly short sentences and irrelevant numbers [25]; output filtering uses discriminators or classifiers [21].Source: §3.3 · p. 24

Overview

• The section compares DA by task because differing datasets, metrics, architectures, and experimental settings make direct cross-method performance comparisons meaningless.The analysis covers text classification, text generation, and structured prediction.Source: §4 · p. 25

Text classification

• DA methods are applied most widely in text classification because its direct label-prediction format makes semantic preservation relatively simple.Each individual DA method could be applied to text classification.Source: §4 · p. 26

Text generation and structured prediction

• Text generation favors sampling-based methods for semantic diversity, whereas structured prediction favors paraphrasing-based methods because it is sensitive to output format.Structured prediction therefore has higher requirements for data validity.Source: §4 · p. 26

DA methods

• Simple unsupervised methods such as machine translation, thesaurus-based paraphrasing, and random substitution are popular, while learnable paraphrasing and sampling methods attract attention for diversity and effectiveness.The learnable methods include paraphrasing-based model generation and sampling-based pretrained models.Source: §4 · p. 26

Development status

• DA applications have increased over the years, with text classification adopting them first and most often, while text generation and structured prediction receive increasing attention.The timeline in Table 3 records DA methods across the three task types.Source: §4 · p. 26, Table 3 · p. 27
• Sampling-based methods have recently gained momentum in text classification and text generation, whereas structured prediction still tends toward paraphrasing- and noising-based methods.Sampling-based methods bring more gains to powerful pretrained language models than paraphrasing-based methods in text classification and text generation.Source: §4 · p. 26

5.1. Pretrained Language Models

• Pretrained language models implicitly add training data through self-supervised learning, whereas task-related augmentation targets specific tasks and may avoid out-of-domain data.PLMs mine supervision from large-scale unsupervised data, which can be out-of-domain for downstream tasks.Source: §5.1 · p. 29

5.2. Contrastive Learning

• Contrastive learning uses data augmentation to create same-label similar samples, then selects random instances as negatives to learn discriminative embeddings.Its objective brings similar samples closer and pushes dissimilar samples farther apart.Source: §5.2 · p. 29

5.3. Other Data Manipulation Methods

• Other data-manipulation methods differ from augmentation: oversampling resamples minority data, cleaning improves quality, weighting changes sample importance, and synthesis creates complete labeled examples.These methods generally do not generate augmented data in the same way as model- or rule-based augmentation.Source: §5.3 · p. 29

5.4. Generative Adversarial Networks

• GANs use a generative model to challenge a discriminator, while some augmentation methods directly use generative models to create training data; GAN-based augmentation has proven effective in several settings [62] [95].GANs were introduced by Goodfellow et al. [137].Source: §5.4 · p. 29

5.5. Adversarial Attacks

• Adversarial attacks generate examples intended to cause model errors, and some work uses augmentation such as code-switch substitution to create such examples for consistency regularization.The examples are generated by attacking a machine-learning model.Source: §5.5 · p. 30

Theoretical Narrative

• NLP data augmentation lacks systematic probing and theoretical analysis explaining its mechanisms beyond downstream effectiveness, especially given natural language’s discrete nature.Related image-domain theories view augmentation as encoding invariance, reducing variance, or regularizing models.Source: §6 · p. 30

More Exploration on Pretrained Language Models

• Pretrained-language-model augmentation remains limited to [MASK] completion, fine-tuned generation [44], and self-training, motivating more diverse, higher-quality, lower-cost approaches [144].The section poses whether augmentation remains helpful in the pretrained-model era and how pretrained knowledge can support generation.Source: §6 · p. 30

Few-shot Scenarios

• Few-shot augmentation relies mainly on paraphrasing, which preserves validity but limits semantic diversity as pretrained models already encode much of its knowledge.The resulting opportunity is to generate high-quality augmented data that enables fast generalization and problem solving under severe data scarcity.Source: §6 · pp. 30–31

Retrieval Augmentation

• Retrieval augmentation improves parameter efficiency and domain adaptation, while future systems could retrieve common-sense, factual, linguistic, and structured knowledge [147] [148] [149] [150].Structured retrieval could incorporate knowledge-base population and open information extraction.Source: §6 · p. 31

More Generalized Methods for NLP

• Because NLP is discrete and includes task-specific structured prediction, no augmentation method currently works across all NLP tasks; unified task formulations may enable more general methods.T5, GPT3, and prompting suggest that natural-language task formalization can transcend traditional task categories.Source: §6 · p. 31

Working with Long Texts and Low Resources Languages

• Augmentation for long texts remains limited to simple paraphrasing and noising, while low-resource languages lack sufficient methods despite greater need [29].Both directions remain underexplored and promising, alongside existing progress on short texts and common languages.Source: §6 · p. 31

• The survey categorizes NLP data augmentation methods by augmented-data diversity into paraphrasing, noising, and sampling, helping researchers understand and develop DA methods.Source: §7 · p. 32
• It describes DA characteristics, applications across NLP tasks, and their development through a timeline.Source: §7 · p. 32
• The paper provides practical tricks and strategies for researchers and practitioners seeking better model performance.Source: §7 · p. 32
• It distinguishes data augmentation from related topics and outlines current challenges and opportunities for future research.Source: §7 · p. 32

• Resources include a visual survey and repositories for EDA, unsupervised data augmentation, PyTorch implementations, and nlpaug.These resources provide information or APIs for data augmentation.Source: Appendix A · p. 55
• Additional repositories cover adversarial examples, multimodal augmentation, and collaborative natural-language transformation resources.The listed resources are TextAttack, AugLy, and NL-Augmenter.Source: Appendix A · p. 55
• The appendix includes a Turkish nlpaug resource and Chinese resources using EDA, BERT, BART, back-translation, and PaddleNLP.The Chinese resources include a customized PaddleNLP repository and Tencent AI Lab’s embedding corpus for Chinese words and phrases.Source: Appendix A · p. 55