Image Data Augmentation for Deep Learning: A Survey

1 Introduction

• Applying augmentation across computer-vision tasks is inefficient and poorly scalable because image operations must also transform task-specific labels.The passage notes that methods for object detection cannot be directly applied to semantic segmentation.Source: §1 · p. 1
• The field lacks theoretical guidance on sufficient dataset size, while generating qualified data from very small original datasets remains challenging.Generated-data quantities are typically chosen through personal experience and extensive experiments.Source: §1 · p. 1
• Earlier reviews were limited in task scope or coverage: focused on classification, while lacked evaluation across actual tasks and newer methods.Other related reviews focused on face recognition or did not systematically review different approaches.Source: §1 · pp. 1–2
• The paper proposes a taxonomy, reviews augmentation for object detection, semantic segmentation, and classification, and evaluates methods and combinations across models and public datasets.It also discusses future research directions and organizes methods into traditional manipulation, erasing, mixing, auto-augmentation, feature augmentation, and deep generative approaches.Source: §1 · p. 2

2.1 Image Manipulation

• Basic manipulations such as rotation, flipping, and cropping directly transform images and are generally easy to implement.Their use assumes existing data is distributed close to actual data; translation and rotation can move image areas beyond the boundary, requiring interpolation that commonly fills blanks with black, while objects of interest should remain in frame.Source: §2.1 · p. 2

2.2 Image Erasing

• Erasing methods mask image regions: Cutout [DeVries and Taylor, 2017b] and HaS [Singh et al., 2018] hide patches, while GridMask [Chen et al., 2020] distributes squares uniformly.Random erasing replaces a random rectangle with random values; GridMask controls mask density and size, and FenceMask [Li et al., 2020] balances object occlusion with information retention. Cutout is described as improving CNN robustness and overall performance; HaS prompts networks to use other relevant content when discriminative content is hidden.Source: §2.2 · p. 2

2.3 Image Mix

• Pairing samples [Inoue, 2018] average two images pixelwise, whereas Mixup [Zhang et al., 2017] convexly combines sample pairs and labels.Mixup links augmentation linearly to the supervision signal and regularizes networks toward simple linear behavior between training samples.Source: §2.3 · p. 2
• CutMix [Yun et al., 2019] replaces removed regions with another image’s patch, while Fmix [Harris et al., 2020] uses random masks thresholded from low-frequency Fourier samples.Fmix supports varied mask shapes and is reported to improve performance over Mixup and CutMix; AugMix [Hendrycks et al., 2019] convexly combines results from chains of augmentation operations applied to the same image, while ManifoldMix [Verma et al., 2019] mixes hidden representations to improve representations and decision boundaries across layers.Source: §2.3 · p. 3

3.1 Auto Augment

• AutoAugment searches policies of augmentation operations and magnitudes for the highest validation accuracy, replacing manual method design with automated policy selection.Its search algorithm selects from a search space of candidate policies because different data can benefit from different augmentations.Source: §3.1 · p. 3
• Fast AutoAugment uses density matching, PBA searches nonstationary schedules, and RandAugment reduces search space while improving on earlier automated methods.The passage reports that Fast AutoAugment speeds search relative to AutoAugment, PBA matches its performance on multiple datasets with less computation, and RandAugment surpasses previous automated techniques, including AutoAugment and PBA.Source: §3.1 · p. 3
• KeepAugment preserves salient image regions, Augmentation-wise Weight Sharing improves search efficiency, and OHL-Auto-Aug jointly optimizes policy-distribution and network parameters.Augmentation-wise Weight Sharing uses the greater influence of later-training augmentations to make direct search on large datasets affordable; OHL-Auto-Aug optimizes its parameterized augmentation distribution jointly with network parameters.Source: §3.1 · p. 3

3.2 Feature Augmentation

• Feature augmentation transforms learned representations through noise, interpolation, extrapolation, prototypical information, or exchanged latent-feature moments.FeatMatch uses within-class and across-class prototypes for varied complex transformations; Moment Exchange replaces one image’s learned-feature moments with another’s.Source: §3.2 · p. 4

3.3 Deep Generative Models

• GANs can generate images while discriminators constrain their gap from original images, but generating and evaluating high-quality augmented data remains challenging.The section presents GANs as a way to augment data through unsupervised image generation.Source: §3.3 · p. 4
• Pix2Pix requires paired data, CycleGAN translates unpaired domains but needs n×(n−1) models for n domains, and StarGAN uses one model for multiple domains.StarGAN v2 addresses deterministic outputs by generating diverse target-domain images using domain and style distinctions.Source: §3.3 · p. 4

4.1 Semantic Segmentation

• Data augmentation improves mean IoU for all evaluated semantic segmentation models.Experiments use PASCAL VOC and compare deeplabv3+, PSPNet, GCNet, and ISANet with and without manipulation, erasing, and mix methods.Source: §4.1 · p. 4, Table 2 · p. 4

4.2 Image Classification

• Data augmentation leads to average accuracy improvement (AAI) in image classification.The experiments compare augmented and unaugmented Wide-ResNet, DenseNet, and Shake ResNet on CIFAR-10, CIFAR-100, and SVHN.Source: §4.2 · p. 5, Table 3 · p. 5

4.3 Object Detection

• On COCO2017, augmentation methods from section 4.1 are evaluated with and without augmentation using FasterRCNN and CenterNet; average mAP, AP50, and AP75 show promising performance improvements.Source: §4.3 · p. 5

Theoretical Research on Data Augmentation

• Theoretical support is needed to explain why augmentation methods work and how much training data is sufficient.Pairing samples and mixup can improve accuracy despite unclear reasons, while dataset sizes are often chosen through experience and extensive experiments.Source: §5 · p. 5

The Evaluation of Data Augmentation Methods

• No unified metric evaluates synthetic-image quality, leaving human judgments and task-specific performance measures as imperfect alternatives.Human evaluation, including AMT, is time-consuming, labor-intensive, and subjective; task metrics do not assess synthetic data itself, so metrics should address data diversity and distribution consistency. The passage cites [Salimans et al., 2016] on the open evaluation problem.Source: §5 · p. 5

Class Imbalance

• Minority-class oversampling can cause saturation and overfitting, so generated data should preserve diversity while approximating the training distribution.Class imbalance skews learning toward majority examples, leaving minority classes poorly modeled; SMOTE oversamples minority classes by repeatedly drawing from the current dataset.Source: §5 · pp. 5–6

The Number of Generated Data

• The amount of generated data needed is unresolved because increasing training-set size does not always improve performance.Performance may plateau when added data do not increase diversity.Source: §5 · p. 6

The Selection and Combination of Data Augmentation

• Augmentation methods must be carefully selected and combined for each new task and dataset.Combined methods have sometimes outperformed a single method, but suitable methods differ across datasets and tasks and must be designed, implemented, and tested accordingly.Source: §5 · p. 6

• Data augmentation is presented as an effective solution to the shortage of limited labeled image data.Source: §6 · p. 6
• The paper comprehensively reviews image data augmentation methods across computer vision tasks and proposes a taxonomy of representative approaches.Source: §6 · p. 6
• The paper empirically compares augmentation methods across computer vision tasks and discusses challenges and future perspectives.Source: §6 · p. 6