PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Data Augmentation for Text Classification

Markus Bayer, Marc-André Kaufhold, Christian Reuter

arXiv:2107.03158v6cs.CLcs.AI

TL;DR

Textual data augmentation lacks universal transformations that preserve labeling quality, and more data alone does not necessarily solve learning problems. This survey organizes augmentation methods for textual classification and identifies research directions for improving evaluation and adapting methods to large pre-trained language models.

  • Problem

    NLP research lacks universal automatically applied text transformations that maintain labeling quality, while increasing training data alone does not necessarily solve the learning problem.

  • Method

    The survey clusters augmentation methods using a high-level data-structure-based taxonomy, then subdivides them into finer-grained groups ranging from noise induction to generating entirely new instances.

  • Results

    The survey provides an overview of textual data augmentation approaches and identifies research directions involving large pre-trained language models, improved approaches, and more comprehensive evaluation.

  • Takeaways & Limitations

    Data augmentation can support regularization, reduce labeling effort, and lower reliance on real-world data, while future work should account for transfer learning's effects on existing methods.

  • Takeaways & Limitations

    Many augmentation methods produce high-quality augmented data only when the original amount of training data is sufficiently large.

Abstract

from arXiv · show

Data augmentation, the artificial creation of training data for machine learning by transformations, is a widely studied research field across machine learning disciplines. While it is useful for increasing a model's generalization capabilities, it can also address many other challenges and problems, from overcoming a limited amount of training data, to regularizing the objective, to limiting the amount data used to protect privacy. Based on a precise description of the goals and applications of data augmentation and a taxonomy for existing works, this survey is concerned with data augmentation methods for textual classification and aims to provide a concise and comprehensive overview for researchers and practitioners. Derived from the taxonomy, we divide more than 100 methods into 12 different groupings and give state-of-the-art references expounding which methods are highly promising by relating them to each other. Finally, research perspectives that may constitute a building block for future work are provided.

1 INTRODUCTION

Textual data augmentation addresses limited training data and diverse application goals, but NLP must preserve labeling quality under automatic transformations. This survey organizes the field, relates methods to models and contexts, and identifies future research directions.

  • Training-data quantity remains decisive for supervised classifier quality, motivating artificial data creation through transformations.
  • NLP data augmentation must establish transformations that automatically maintain labeling quality, unlike simpler invariance assumptions in image and speech augmentation.
  • Textual augmentation supports goals including low-data training, class balancing, and robustness against adversarial examples.
  • The survey highlights goals and applications, a comprehensive text-classification overview, and a data-structure-driven taxonomy with fine-grained method groups.
  • It explains more than 100 approaches in 12 groups, compares their conception and performance across models and contexts, and proposes future research opportunities.

2 BACKGROUND: FOUNDATIONS, GOALS, AND APPLICATIONS OF DATA AUGMENTATION

Data augmentation artificially enlarges training data to address data scarcity, labeling costs, generalization, regularization, privacy, and robustness. Its usefulness depends on the task and on transformations preserving class information with sufficient safety.

  • Data augmentation addresses insufficient training data by artificially enlarging the available training set through transformations.
  • Label preservation means that a transformation preserves class information, while safety captures the probability that the correct class is maintained.
  • Augmentation can reduce labeling effort, support class balancing, and improve classifier robustness in classification problems.
  • Data augmentation can also function as regularization by promoting generalization without degrading representational capacity or retuning other hyperparameters.
  • Text transformations are difficult because small edits can change sentiment or other class information, and augmentation cannot be effectively applied in every classification problem.

3 TEXTUAL DATA AUGMENTATION METHODS

The survey focuses on textual classification augmentation methods, organizing them through a method-oriented and data-structure-driven taxonomy. It presents grouped explanations, tabular comparisons, performance information, and context-sensitive analysis for researchers and practitioners.

  • The survey focuses on augmentation methods for topic classification, sentiment analysis, and spam identification, while other NLP tasks receive non-comprehensive treatment.
  • Methods are categorized from a data-structure-driven, high-level perspective and subdivided into fine-grained groups.
  • The survey compares augmentation methods by conception and performance while accounting for underlying models and application contexts.
  • Grouped descriptions explain method differences and results, while tables summarize similar approaches and extract improvement information.
  • The overview is intended to help readers identify promising techniques and relevant use-case specifics, alongside future research directions.

3.1 Data Space

Data-space augmentation transforms raw input data into readable textual form.

  • Data-space augmentation transforms the input data in its raw form into readable text.

3.1.1 Character Level

Character-level augmentation introduces small textual perturbations, including artificial or natural spelling noise, adversarial letter changes, and rule-based surface transformations. Reported effects range from improved robustness and classification accuracy to degraded performance, depending on the transformation and evaluation setting.

  • Noise Induction: Character-level noise randomly switches, scrambles, deletes, inserts, or replaces letters, while natural noise uses common spelling mistakes.These transformations are applied to training text to expose models to altered surface forms.
  • Noise Induction: 0.62% additional accuracy was the best reported improvement for a CharCNN-LSTM on AG News, while another method improved accuracy by 0.24 points and one decreased it by 0.33 points.The comparison used basic character-noise methods and shows that outcomes vary across approaches.
  • Evaluation Caveat: Character-level evaluations using basic architectures and no embeddings require further studies to validate usefulness in current settings.The reported 2.5% improvement was measured against the best-functioning baseline under that restricted evaluation setup.
  • Noise Induction: Noise operations made neural machine translation models more robust against adversarial attacks, but natural noise almost consistently worsened translation performance against the baseline.Robustness and baseline quality therefore did not move uniformly together.
  • Noise Induction: Spelling-error augmentation improved the best XGBoost baseline by an additional 1.5% and helped classifiers handle spelling mistakes in prospective texts.The passage identifies social-network text as a potential application setting.
  • Rule-based Transformations: Rule-based transformations can preserve meaning, but their rules are difficult to establish because grammar and language dependencies constrain valid surface changes.An English short-form transformation such as “I am” ↔ “I’m” produced a 0.5% accuracy gain for the best XGBoost baseline.

3.1.2 Word Level

Word-level augmentation replaces, inserts, masks, drops, or perturbs words using noise, synonyms, embeddings, or language models. The survey reports gains for several methods but also highlights label-preservation problems and mixed results across tasks and settings.

  • Noise Induction: Word-level noise includes word replacement, unigram noising, blank noising, word dropout, random deletion, and random swapping.These methods alter lexical content to create additional training instances or noise patterns.
  • Noise Induction: A combination of Li et al.’s noise methods promised up to 1.7 points of accuracy improvement.The methods combine syntactic noise, semantic synonym noise, and word dropout.
  • Noise Induction: Random swap and deletion can decrease classification scores because they are not label preserving for some sentiment examples.The survey gives a negation-sensitive sentence transformation as an example of an altered sentiment label.
  • Noise Induction: Padding-shift noise produced additive gains of more than 5.8% Macro-F1 on a hate-speech dataset, although another meaningless-word insertion method was not evaluated in isolation.The padding-shift method moves instances within their existing padding rather than changing their lexical content.
  • Synonym Replacement: Synonym substitution replaces selected words with thesaurus-based synonyms, with approaches differing in target-word selection, synonym selection, databases, and probability rules.Some methods exclude stop words or selected POS tags, while EDA can randomly insert synonyms instead of replacing specific words.
  • Embedding Replacement: Embedding replacement achieved an additive improvement of up to 2.4 points in F1-Score with logistic regression, while adversarial training reported no test-set improvement but a safer model.The survey also notes that counter-fitting offers fewer replacement possibilities because downstream-task embeddings reduce corpus coverage.
  • Embedding Replacement: Embedding replacement selects context-fitting words intended to preserve the text’s basic substance, but contextual meaning is not guaranteed to remain unchanged.Meaning distortions can consequently change labels, such as reversing sentiment between “fantastic” and “horrible.”
  • Replacement by Language Models: Language-model replacement methods, including c-BERT, increased classifier performance across evaluated tasks and significantly outperformed the original c-BERT approach in a low-data regime.The fixed language model may no longer preserve labels in low-data settings.

3.1.3 Phrase and Sentence Level

Phrase- and sentence-level methods manipulate grammatical structures, semantic entities, or labeled substructures to create new examples. Reported benefits include improved generalization, low-resource performance, and few-shot gains, but some methods depend on language-specific structures or short texts.

  • Structure-based Transformation: Structure-based augmentation uses grammatical features such as dependency or constituent grammars and POS tags to modify text structures.These approaches are consequently more limited to particular languages or tasks.
  • Structure-based Transformation: Cropping shortens sentences around subjects and objects, while rotation moves flexible fragments; both methods were reported as suitable for many low-resource languages.The authors caution that rotation may generate noise in English because it depends on language-specific grammatical structures.
  • Semantic Transformation: Semantic Text Exchange replaces identified phrases with similar phrases generated using an attention-based language model while attempting to preserve fluency and sentiment.A Yelp adaptation selected frequent nouns as replacement entities and split the dataset into windows because the method suits short texts.
  • Structure-based Transformation: Inversion and passivation produced higher generalization capability in natural language inference.The survey relates this result to insufficient MNLI examples demonstrating the necessity of syntax, which limited BERT’s use of syntactic information.
  • Interpolation: SUB² substitutes labeled substructures under rules requiring compatible span lengths, phrase types, constituency labels, and class labels.Its classification variant nearly doubled accuracy on subsamples of SST-2 and AG News and outperformed c-BERT.
  • Structure-based Transformation: A lexicalized probabilistic context-free grammar method combines or substitutes grammar trees within and across same-class sentences, improving performance in few-shot semi-supervised learning.Words can be replaced by same-POS words from other sentences of the same class or by WordNet synonyms.

3.1.4 Document Level

Document-level augmentation methods generate new texts through round-trip translation or generative models. The survey describes their mechanisms, filtering and sampling choices, and reported classification outcomes.

  • 3.1.4.1 Round-trip Translation: Round-trip translation converts text into another language and back to the source language to create paraphrases.Translation variability can change terms or sentence structure, producing alternative textual forms.
  • 3.1.4.1 Round-trip Translation: Round-trip approaches differ in language choices and filtering methods, with random sampling and temperature used to increase diversity.Filtering is important because twofold translation may be faulty.
  • 3.1.4.2 Generative Methods: Unconditional VAEs sampled from the prior distribution achieve improvements of up to 2 F1-points.Prior sampling produces highly diverse instances, while posterior sampling yields text semantically closer to the training data.
  • 3.1.4.2 Generative Methods: Conditional VAEs with reconstruction perform best in intrinsic and extrinsic evaluations, whereas the discriminator variant performs poorly with limited data across many classes.The reported weakness is attributed to the small amount of available training data for the number of classes.
  • 3.1.4.2 Generative Methods: Other generative methods report strong or consistent gains, including superiority to EDA, round-trip translation, and Tmix, but large pretrained generators may replicate omitted training or test instances.The latter concern is raised for GPT-3-based augmentation applied to scarce datasets.

3.2 Feature Space

Feature-space augmentation transforms input representations rather than raw text, including noise injection, adversarial training, and interpolation. Reported benefits vary by method and setting, with several approaches improving robustness or benchmark performance while some gains remain limited or scope-dependent.

  • Feature-space augmentation transforms input feature representations, such as neural activation vectors, encoder-decoder encodings, or LSTM hidden states.
  • Feature-space methods are not uniformly effective: in few-shot settings all methods are highly beneficial, whereas in other settings they only slightly improve classification results.
  • Adversarial training: Adversarial training adds perturbation noise within a norm ball and optimizes model parameters against perturbations that increase loss.
  • Adversarial training: Several adversarial methods improve robustness or benchmark performance, including FreeLB, which consistently exceeds the baseline and two other methods across several tasks.
  • Adversarial training: ALUM improves RoBERTa by + 0.7 on MNLI, while standard continual pretraining adds no gains, and it significantly improves robustness across three adversarial datasets.
  • Interpolation: Interpolation methods mix feature representations or instances, and reported results include reduced overfitting, less-specific decision boundaries, and improved GLUE or MNLI performance.
  • Interpolation: Some interpolation methods have scope limits, including unsuitability for sequence tagging, while feature-space evaluation results are described as promising for textual applications.

3.3 Combination of Augmentation Methods

Combining augmentation methods can diversify training instances and improve results, but the effectiveness of combinations depends on the methods, task, and whether stacking is feasible.

  • Combining separate or stacked augmentation methods is used to create more diversified training instances.
  • Word-level and phrase-level methods produced very good results when combined, and other combinations improved in-domain evaluations.
  • Round-trip translation, random swap, random deletion, and random synonym insertion produced the best improvement for an LSMT classifier when combined.
  • Stacking is not always feasible because feature-space methods generally cannot be applied before data-space methods.
  • In tested combinations, round-trip translation with adversarial examples produced the best results, while another stack reduced minimal error in two out of three evaluation settings.
  • A GAN can learn sensible sequences of predefined transformations and significantly improve performance on a relation extraction task using language-model replacements.

3.4 Training Strategies

Training strategies use augmentation to enforce consistency, alter example order, or support contrastive objectives. Reported gains are strongest in some low-data and pretrained-model settings, but no single augmentation method is uniformly best.

  • Consistency training: Consistency training encourages identical predictions for original and label-preserving augmented instances, allowing models to learn invariances.
  • Consistency training: Consistency training with round-trip translation and TF-IDF replacement improved accuracy by 22.79% in a BERT-based low-data Amazon-2 regime.
  • Consistency training: The same approach outperformed the state of the art on IMDb using only 20 supervised instances.
  • Contrastive learning: Contrastive learning brings original and augmented instances closer while separating other instances, and several augmentation strategies can support this objective.
  • Contrastive learning: Across evaluated tasks, there is no clear best augmentation method for contrastive learning.
  • Consistency training: Combining adversarial training with round-trip translation and consistency training achieved further improvements in supervised settings.
  • Training order: Reordering training examples can correct unfavorable behavior learned from noisy augmented data, and this strategy outperformed importance-weight loss in the reported comparison.

3.5 Filtering Mechanisms

Filtering mechanisms select augmented instances based on similarity, validation-loss effects, diversity, or class quality. They help manage harmful or low-quality examples, but some filters reduce sample diversity or require human judgment.

  • Filtering is especially important for augmentation methods that do not perfectly preserve labels.
  • Similarity filtering: Similarity-based filters compare generated instances with originals using unigram overlap or metrics such as Levenshtein distance, Jaccard similarity, and Hamming distance.
  • Classifier filtering: Classifier-based filtering can significantly reduce sample diversity because the classifier is already familiar with the class data.
  • Quality filtering: Embedding-based quality filtering incorporates a human expert to determine the correct threshold for class-related generated examples.
  • Loss-based filtering: Validation-loss filtering identifies detrimental generated instances by measuring whether adding each instance increases validation loss, but this would require retraining for each example.
  • Diversity filtering: Diversity-oriented filtering selects examples that maximize the number of unique unigrams.

4 DISCUSSION: A RESEARCH AGENDA FOR TEXTUAL DATA AUGMENTATION

The survey develops a research agenda around comparing augmentation methods, adapting them to large pre-trained models, improving label preservation and novelty, and making evaluation and application more reliable.

  • Research agenda: Reported results are restricted to particular models and datasets, limiting direct comparison across augmentation methods.The survey therefore frames standardized comparison as a central research need.
  • Large pre-trained models: Large pre-trained language models can make some augmentation methods obsolete because they pursue similar goals.The survey calls for studying augmentation methods specifically in the context of transfer learning.
  • Large pre-trained models: Random, slight transformations tend to be less beneficial with large pre-trained models than adversarial, cutoff, interpolation, and some generative methods.Embedding- and language-model-based replacement methods can also improve results in combination with these models.
  • Novelty and label preservation: The survey questions whether unseen linguistic patterns are always necessary, because SUB² achieves high gains through phrase interpolation from training data.This contrasts with the hypothesis that augmentation helps only when it introduces new linguistic patterns.
  • Novelty and label preservation: Conditional generative models replicate data classes best for relatively narrow task definitions, while broad-topic tasks appear less suitable.New conditioning methods and filtering mechanisms are identified as possible ways to mitigate this limitation.
  • Evaluation and usability: Feature-space adversarial methods require evaluation of whether transformations preserve labels, since nearby natural-language inputs do not necessarily share labels.The survey also proposes inspecting interpolated data-space instances to study their diversity.

5 CONCLUSION

The survey synthesizes textual data augmentation methods, their goals, and their limitations, then proposes an agenda for better evaluation, understanding, and usability. It emphasizes that transfer learning changes which methods remain useful, while augmentation can still support several practical objectives.

  • Conclusion: The survey covers textual data augmentation methods applied in feature and data space, from noise induction to generating completely new instances.It organizes the methods into high-level and fine-grained groups.
  • Conclusion: Data augmentation can support regularization, reduce labeling effort, lower real-world data usage, address privacy-sensitive settings, balance datasets, and improve robustness against adversarial attacks.These applications are presented as distinct goals of textual augmentation.
  • Conclusion: Transfer learning can make methods with goals similar to pre-trained models obsolete, motivating approaches that introduce linguistic patterns unseen during pre-training.This conclusion follows the survey's discussion of large pre-trained language models.
  • Conclusion: Many methods require sufficient original data, cannot cover all transformations or remove original biases, and may introduce undesirable biases.For example, augmentation is unlikely to create sports articles when the source news dataset contains none.
  • Conclusion: Data augmentation can be time-consuming and resource-intensive, making some methods infeasible for time-critical development and increasing the costs of generative-model training.The survey identifies crisis informatics as an example of a time-sensitive domain.
  • Conclusion: The proposed research agenda targets large pre-trained models, improved methods, comprehensive comparison standards, deeper understanding, and greater application usability.The agenda is presented as a way to mitigate limitations and amplify augmentation's strengths.
Loading 2107.03158v6…