PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Survey on Data Augmentation

Zaitian Wang, Pengfei Wang, Kunpeng Liu, Pengyang Wang, Yanjie Fu, Chang-Tien Lu, Charu C. Aggarwal, Jian Pei, Yuanchun Zhou

arXiv:2405.09591v4cs.LGcs.AI

TL;DR

Data augmentation lacks a unified cross-modality account despite its use for scarce or imbalanced data. This survey synthesizes five modalities with a data-centric taxonomy based on information sources, highlighting shared operations and evaluation considerations.

  • Problem

    Existing surveys lack a comprehensive account spanning all data types and use incompatible modality-specific or operation-centric taxonomies.

  • Method

    The survey proposes a modality-independent taxonomy organized by where augmented information comes from and applies it across five modalities.

  • Results

    Across modalities, methods share operations such as masking or noise injection and structural cropping, enabling conceptual transfer between underexplored modalities.

  • Takeaways & Limitations

    Augmentation effectiveness is best understood using both affinity and diversity rather than downstream performance alone.

Abstract

from arXiv · show

Data augmentation is a series of techniques that generate high-quality artificial data by manipulating existing data samples. By leveraging data augmentation techniques, AI models can achieve significantly improved applicability in tasks involving scarce or imbalanced datasets, thereby substantially enhancing AI models' generalization capabilities. Existing literature surveys only focus on a certain type of specific modality data and categorize these methods from modality-specific and operation-centric perspectives, which lacks a consistent summary of data augmentation methods across multiple modalities and limits the comprehension of how existing data samples serve the data augmentation process. To bridge this gap, this survey proposes a more enlightening taxonomy that encompasses data augmentation techniques for different common data modalities by investigating how to take advantage of the intrinsic relationship between and within instances. Additionally, it categorizes data augmentation methods across five data modalities through a unified inductive approach.

I. INTRODUCTION

Data augmentation addresses data scarcity and imbalance by creating modified copies of existing samples, while existing surveys remain fragmented by modality and operation. This survey instead develops a unified data-centric taxonomy spanning five modalities.

  • Data scarcity, difficult collection, and labor-intensive labeling motivate augmenting existing datasets with artificial samples.
  • B. Existing Surveys: Existing surveys mainly focus on individual modalities or operation-centric categories, limiting a unified view of augmentation mechanisms.
  • The survey proposes a modality-independent, data-centric taxonomy covering image, text, graph, tabular, and time-series data.
  • B. Existing Surveys: Across prior work, image augmentation was studied earliest, followed by surveys addressing text, graph, tabular, and time-series augmentation separately.
  • B. Existing Surveys: A comprehensive cross-modality survey can expose shared patterns such as altering sample features or mixing values between samples.

III. TAXONOMY

The taxonomy classifies augmentation by how many source samples are used and whether value or structural information is transformed. It distinguishes single-instance, multi-instance, and dataset-level generation, including vanilla and exogenous generation.

  • The first taxonomy tier separates single-instance, multi-instance, and dataset-level augmentation according to the number of source samples used.
  • Single-instance augmentation perturbs one sample, while multi-instance augmentation combines multiple samples through operations such as interpolation or concatenation.
  • The second tier distinguishes value-based transformations from structure-based transformations according to which information is perturbed.
  • Multi-instance methods include value-based mixture, which arithmetically mixes values, and structure-based combination, which pieces data or parts together.
  • Dataset-level augmentation learns the dataset distribution to generate new samples, classified as vanilla generation or exogenous generation using external resources.

IV. SINGLE-INSTANCE LEVEL AUGMENTATION

Single-instance augmentation perturbs one sample’s values or structure, with modality-specific elements and relationships determining the available operations. Image examples include masking, appearance changes, and structural manipulation.

  • Single-instance methods modify one sample’s elemental values or structural relationships to produce augmented data.
  • Images represent value information through pixel colors and structure through spatial relationships among pixels.
  • Pixel erasing masks image regions, whereas photometric transformation changes overall color or style while preserving essential information.
  • Table II summarizes augmentation methods with their categories, operations, and corresponding analyses.

2) Wording Perturbation:

Wording perturbation changes tokens through replacement, addition, or deletion, while related modality-specific methods perturb graph attributes, tabular cells, and time-series values or decompositions. These operations illustrate value-oriented single-instance augmentation across modalities.

  • Wording Perturbation:: Token replacement, addition, and deletion perform word-level manipulation for text augmentation.
  • Wording Perturbation:: Token replacement substitutes selected words with alternative wording such as synonyms.
  • Wording Perturbation:: Token addition inserts noise or informative content, including random synonyms or punctuation marks.
  • Wording Perturbation:: Token deletion removes selected words or replaces them with blank placeholders to introduce noise and reduce overfitting.
  • Graph, tabular, and time-series augmentation perturb attributes, cell values, amplitudes, phases, or decomposed components through modality-specific operations.

B. Structure-based Transformation

Structure-based transformation augments data by perturbing relationships within instances, across instances, or across graph topology. The surveyed operations include spatial, syntactic, and graph-structural transformations, with topology perturbation requiring caution when crucial connections are altered.

  • B. Structure-based Transformation: Structure-based transformations perturb relationships among elements rather than only changing their individual values.The section covers spatial relationships in images, syntactic relationships in text, and topology in graphs.
  • B. Structure-based Transformation: Image cropping selects regions, while geometric transformation changes pixel positions through operations such as flipping, rotation, and translation.Cropping removes other areas; geometric transformations preserve pixel values while changing spatial relationships.
  • B. Structure-based Transformation: Text structure can be transformed through sentence cropping, which preserves local syntactic cues, and sentence morphing, which reorders words or grammatical components.Sentence cropping uses dependency relationships, whereas morphing includes swaps, inversion, and passivization.
  • B. Structure-based Transformation: Graph topology augmentation modifies nodes or edges, extracts subgraphs, diffuses information, or rewires connections to create structurally varied graphs.Graph diffusion uses higher-order neighborhood information, while rewiring changes edge assignments.
  • B. Structure-based Transformation: Topology perturbation can remove crucial edges, impair connectivity, and introduce inter-class links that may worsen representation collapse.These risks constrain graph algorithms requiring fully connected inputs and motivate careful use of topology perturbation.

4) Table Subsetting:

Table subsetting creates augmented tabular or visual samples by retaining subsets of features or content and requiring models to recover the omitted information. Related policy-based and prompt-based methods instead search for or generate controlled transformations.

  • 4) Table Subsetting:: Table subsetting divides columns into feature subsets and trains models to reconstruct all features from incomplete views.SubTab forces representations to be learned from only a percentage of the original features.
  • 4) Table Subsetting:: Window slicing extracts temporal segments as new samples, while sequence morphing stretches, compresses, or rearranges those segments.Each extracted slice receives the original sequence label, and morphing changes local timing or order.
  • 4) Table Subsetting:: Policy-based augmentation searches for operation sequences with selected probabilities and magnitudes, using downstream performance to guide policy optimization.AutoAugment uses reinforcement learning, while Fast AutoAugment accelerates the search.
  • 4) Table Subsetting:: Prompt-based image editing conditions generative models on textual descriptions to alter image content, backgrounds, or styles.Methods may combine prompts with reference images and detected edges to generate edited samples.

2) Text Value-structure Manipulation:

Text value-structure manipulation augments both linguistic content and its organization, using operations that preserve or recombine semantic information. The approaches range from hierarchical attention-guided edits to back-translation, paraphrasing, and multi-instance mixing.

  • 2) Text Value-structure Manipulation:: Hierarchical text augmentation uses attention at word and sentence levels to crop and concatenate important components.A hierarchical attention network estimates component roles before selecting augmentation units.
  • 2) Text Value-structure Manipulation:: Back-translation changes wording and syntax through intermediate languages while generally preserving semantic information.The resulting variation is intended to help models learn flexible expressions with the same meaning.
  • 2) Text Value-structure Manipulation:: Paraphrase augmentation asks language models to rewrite sentences while targeting coherence and greater generation diversity.Methods vary prompts, restrict frequently used words, repeat paraphrasing, or train on preference data.
  • 2) Text Value-structure Manipulation:: Multi-instance augmentation combines value and structure information from different samples, with modality-specific operations for images, text, graphs, and other data.For images, mixing may combine colors or patches; for text, it may combine words or sentence fragments.
  • 2) Text Value-structure Manipulation:: Mixup interpolates labeled images and labels, while text adaptations typically interpolate word or sentence embeddings rather than discrete words.Image Mixup operates at the input layer, whereas wordMixup and senMixup adapt the idea to sentence classification.
  • 2) Text Value-structure Manipulation:: Graph propagation augments representations by spreading node features or labels along graph connections.Feature propagation can create views from masked or propagated features, while label propagation extends labels to nearby unlabeled nodes.

4) Tabular Mixup:

Tabular and cross-instance augmentation combines values or structural fragments across samples, while time-series interpolation averages sequences. Image and text patching similarly recombine selected regions or fragments from multiple examples.

  • 4) Tabular Mixup:: SMOTE addresses class imbalance by interpolating minority-class samples using k nearest neighbors in feature space.The survey notes that interpolation in sparse feature spaces can produce unrealistic samples.
  • 4) Tabular Mixup:: Time-series averaging methods generate new sequences by computing weighted or unweighted averages under Dynamic Time Warping.Weighted-DBA allows some sequences to contribute more strongly and can generate infinitely many examples from a given set.
  • 4) Tabular Mixup:: Image patching constructs samples by placing cropped regions from one or more images into another image.RICAP uses four cropped images, CutMix replaces removed regions with patches, and Attentive CutMix selects representative regions.
  • 4) Tabular Mixup:: Text fragment merging replaces sentence segments with fragments drawn from other sentences using syntactic or saliency-based selection.Tree-based methods analyze grammatical roles, while SSMix substitutes low-saliency segments with higher-saliency fragments.
  • 4) Tabular Mixup:: Combining samples across distinct tables is difficult because columns differ in names and representations, motivating methods that exploit partially overlapping columns.TransTab targets transfer and prediction across multiple tables by relaxing fixed-table restrictions.
  • 4) Tabular Mixup:: Neural and prompt-based mixing generate combined images by fusing inputs through learned networks or diffusion models conditioned on textual descriptions.Neural blending uses two input samples, whereas prompt-based methods combine image concepts through text embeddings and denoising.

2) Text Value-structure Mixture by Generative Mixing:

Generative mixing augments data by combining information from multiple samples, including text sentences and graph structures, while preserving soft-label relationships. Dataset-level generators instead learn overall distributions to produce consistent and diverse samples, though graph GANs face mode collapse.

  • 2) Text Value-structure Mixture by Generative Mixing:: Generative mixing combines information from multiple sentences or graph instances, often producing augmented samples with soft labels.GPT3Mix and PromptMix mix sentences, while graph mixup interpolates node attributes, topology, graph representations, or graphons.
  • Dataset-level data augmentation: Dataset-level augmentation learns an overall data distribution and generates new samples that are consistent and diverse in value and structure.Typical generators include autoencoders, GANs, LLMs, and diffusion models.
  • Dataset-level data augmentation: Autoencoders, especially VAEs, can generate augmented tabular datasets that improve downstream task performance, with VAEs providing the most robust result in one comparison.The cited study evaluates four autoencoder types and reports improved task performance for all of them.
  • Generative Adversarial Networks:: GAN-based augmentation spans image, tabular, and time-series data, supporting dataset enlargement, denoising, privacy-oriented synthesis, imbalance handling, and realistic sequence generation.Examples include medical-image augmentation, Table-GAN, CTGAN, RGAN/RCGAN, TimeGAN, WaveGAN, and SpecGAN.
  • Generative Adversarial Networks:: Graph GAN augmentation is limited by mode collapse, in which generators tend to produce non-unique samples.This limits the application of GANs to graph datasets.

3) Large Language Models:

Large language models and diffusion models provide generative augmentation routes across several modalities. LLM augmentation is especially developed for NLP, while diffusion applications to graph and time-series data remain relatively underexplored.

  • 3) Large Language Models:: LLMs form a generative augmentation paradigm, especially for NLP, by synthesizing, refining, or mixing text under label or prompt guidance.Examples include LAMBADA, token-tree generation, GPT-2 reinforcement guidance, GPT3Mix, and PromptMix.
  • 3) Large Language Models:: Multimodal LLM pipelines augment images by enriching textual descriptions or generating image datasets from textual prompts.DIAGen uses GPT-4 descriptions, while another approach uses DALL-E prompts for agricultural imagery.
  • 3) Large Language Models:: LLMs augment tabular data by treating tables or rows as text, generating synthetic records through prompting or feature masking followed by sample selection or reconstruction.CLLM adds curator-based selection, while Pred-LLM converts rows to text and masks features.
  • 3) Large Language Models:: General LLM-based time-series augmentation remains underexplored beyond prediction and sequence expansion, with more activity in audio, voice, and speech.Time-LLM demonstrates predictive ability, while ArzEn-LLM uses code-switched augmentation for multilingual speech recognition.
  • Diffusion Models:: Diffusion models generate realistic and diverse augmented data across image, text, and tabular modalities, while graph and time-series applications are relatively underexplored.Graph diffusion must address discrete graph structures; DiGress uses discrete denoising diffusion for graph generation.
  • Exogenous Information: Exogenous-information augmentation incorporates external datasets or human-learned knowledge, including style transfer, table retrieval, and expert-guided generation.These approaches add external styles, related tabular information, or domain-informed synthetic samples.

2) Leveraging Expert Knowledge:

Expert knowledge augments data by modeling domain structure or future behavior across image, tabular, and time-series modalities. The survey also argues that augmentation should be evaluated through affinity and diversity, not downstream performance alone.

  • 2) Leveraging Expert Knowledge:: Expert knowledge supports image, tabular, and time-series augmentation through computer graphics modeling, relational structure construction, and statistical generation.These examples use object knowledge, learned table relationships, or statistical forecasts to create additional samples.
  • 2) Leveraging Expert Knowledge:: Computer graphics modeling generates diverse, fully labeled images by varying object aspects, poses, and lighting conditions with limited human data-collection effort.The approach uses expert knowledge of how objects appear and computer software to synthesize samples.
  • 2) Leveraging Expert Knowledge:: Relational structure construction addresses tabular data’s limited spatial or relational structure by learning and introducing relationships into the dataset.GOGGLE is presented as a solution for constructing relational structure in tabular augmentation.
  • 2) Leveraging Expert Knowledge:: Statistical generation predicts future time-series values and samples forecast paths to obtain additional data.Examples use LGT and mixture autoregressive models to represent temporal changes.
  • A. Augmentation Evaluation Metrics: Downstream-task performance is an incomplete augmentation criterion because extraneous factors can obscure the augmentation method’s influence on data.The survey identifies affinity and diversity as two critical data-focused metrics.
  • A. Augmentation Evaluation Metrics: Empirical studies report that augmentation is most effective when it achieves both high affinity and high diversity.Affinity concerns semantic consistency and label preservation, whereas diversity reflects breadth of dataset patterns.

B. Selecting the Target

Target selection determines which pixels, words, nodes, or edges are modified during augmentation. Methods increasingly use content, linguistic structure, connectivity, centrality, or similarity to choose informative targets.

  • 1) On Selecting Pixels:: Target selection is a prerequisite for pixel erasing and can range from random regions to object- or region-aware choices.IRE selects areas randomly, ORE restricts erasing to bounding boxes, and RRE adds bounding-box size limits before background erasing.
  • 2) On Selecting Words:: Text selection methods may replace, add, or delete words according to word importance or grammatical role rather than random choice.TEXTFOOLER ranks words by importance, while syntax-aware augmentation uses dependency structure and favors words less central to sentence meaning.
  • 3) On Selecting Nodes and Edges:: Graph augmentation selects target nodes and edges using node centrality, connectivity-preserving strategies, or node similarity.Centrality guides masking, rewiring, and subgraph selection, while similarity guides adding or removing edges.
  • Beyond target selection: Augmented samples can also be used to construct different views for contrastive learning, beyond simply enlarging a training dataset.The cited example applies rotations, flips, blurring, contrast changes, and noise transformations.

D. How to Choose the Appropriate Augmentation Method?

Choosing an augmentation method requires balancing performance, computational cost, label preservation, framework compatibility, and the information retained during augmentation. Cross-modality similarities also suggest that methods and design ideas can transfer between modalities.

  • Method performance depends strongly on task scenarios and nuanced experimental settings, so no universally best augmentation method can be identified.The survey recommends comparing method characteristics and practical requirements under the intended conditions.
  • Advanced augmentation methods may improve performance but can require more time and resources than training on the original dataset.The relevant trade-off is between expected augmentation benefits and the cost of generating augmented data.
  • Augmentation operations must preserve task labels because the same operation can be safe for one dataset but harmful for another.The survey contrasts image rotation for MNIST with token insertion for SST-2 and CoLA.
  • Multi-instance augmentation may conflict with training frameworks that require hard labels or lack embeddings before model training.Mixup can produce soft labels, while hidden-space augmentation requires embeddings generated within the model framework.
  • Cross-modality similarities reveal transferable operations, including masking, structural cropping, policy search, fragment merging, and generative augmentation.Examples span image, text, graph, tabular, and time-series data, motivating adaptation of established ideas to under-explored modalities.
  • Future augmentation methods should account for sample distributions and supervision while exploiting generative models across modalities.The survey identifies informed augmentation and cross-modality use of generative models as concrete directions.
Loading 2405.09591v4…