PaperlayerView exact revision on arXiv

Source-linked AI summary

Image Data Augmentation Approaches: A Comprehensive Survey and Future directions

Teerath Kumar, Alessandra Mileo, Rob Brennan, Malika Bendechache

arXiv:2301.02830v4cs.CVcs.AIcs.LG

TL;DR

Limited labelled data makes deep learning models vulnerable to overfitting and weaker performance on unseen data. This survey develops a taxonomy of advanced image data augmentation, compares its effects across major computer vision tasks, and compiles implementation code. It also identifies unresolved challenges and future research directions.

  • Problem

    Limited labelled data contributes to overfitting, causing models to perform worse on unseen data than on training data.

  • Method

    The survey presents a comprehensive taxonomy of image data augmentation techniques and reviews their applications across computer vision tasks.

  • Results

    The survey compiles and compares augmentation results for image classification, object detection, and semantic segmentation, including supervised and semi-supervised learning.

  • Takeaways & Limitations

    The survey supports deeper understanding, easier comparison, and reproducibility through compiled evaluations and available augmentation code.

Abstract

from arXiv · show

Deep learning (DL) algorithms have shown significant performance in various computer vision tasks. However, having limited labelled data lead to a network overfitting problem, where network performance is bad on unseen data as compared to training data. Consequently, it limits performance improvement. To cope with this problem, various techniques have been proposed such as dropout, normalization and advanced data augmentation. Among these, data augmentation, which aims to enlarge the dataset size by including sample diversity, has been a hot topic in recent times. In this article, we focus on advanced data augmentation techniques. we provide a background of data augmentation, a novel and comprehensive taxonomy of reviewed data augmentation techniques, and the strengths and weaknesses (wherever possible) of each technique. We also provide comprehensive results of the data augmentation effect on three popular computer vision tasks, such as image classification, object detection and semantic segmentation. For results reproducibility, we compiled available codes of all data augmentation techniques. Finally, we discuss the challenges and difficulties, and possible future direction for the research community. We believe, this survey provides several benefits i) readers will understand the data augmentation working mechanism to fix overfitting problems ii) results will save the searching time of the researcher for comparison purposes. iii) Codes of the mentioned data augmentation techniques are available at https://github.com/kmr2017/Advanced-Data-augmentation-codes iv) Future work will spark interest in research community.

I. INTRODUCTION & MOTIVATION

The survey examines image data augmentation as a response to limited labelled data and overfitting in computer vision. It presents a taxonomy, evaluates state-of-the-art techniques across tasks, compiles reproducible code, and identifies open challenges.

  • I. INTRODUCTION & MOTIVATION: Limited data can cause overfitting, where models perform well on training data but poorly on unseen test data.The problem is intensified by privacy constraints and costly, time-consuming human labelling.
  • I. INTRODUCTION & MOTIVATION: Image data augmentation expands datasets by altering sample appearance to provide more diverse views for CNN training.It is presented as a regularization approach addressing data shortages and supporting model generalization.
  • I. INTRODUCTION & MOTIVATION: The survey presents a comprehensive taxonomy and reviews state-of-the-art augmentation techniques with visual examples.The taxonomy separates basic methods, including image manipulation and erasing, from advanced approaches.
  • I. INTRODUCTION & MOTIVATION: Performance is evaluated and compared for several computer vision tasks, while prior surveys were narrower or lacked comprehensive evaluation.Earlier work included studies limited to image classification or specific approaches without a comprehensive taxonomy.
  • I. INTRODUCTION & MOTIVATION: The authors highlight unresolved challenges and identify future directions intended to stimulate further research.The paper explicitly frames future work as an open direction for the research community.
  • I. INTRODUCTION & MOTIVATION: The survey compiles available augmentation code to facilitate result reproducibility and help researchers compare methods more efficiently.The authors also describe improved understanding of augmentation mechanisms and reduced search time as practical benefits.

1) Image Manipulation:

Image manipulation augments training images through positional, geometric, and visual-characteristic changes, with parameter choices constrained by dataset semantics and label integrity.

  • Image manipulation changes an image’s position or pixel values through positional and color manipulations.
  • Geometric Data Augmentation: Geometric augmentation modifies pixel arrangement through rotation, translation, and shearing to diversify training data and improve resilience to transformations.
  • Geometric Data Augmentation: Rotation, translation, and shearing require dataset-aware magnitudes because excessive transformations can change object identity or substantially deform images.Examples include rotating MNIST digits by 180 degrees, shifting an 8 far enough to resemble a 3, or excessive shearing that impairs recognition.
  • Non-Geometric Data Augmentations: Non-geometric augmentation changes visual characteristics rather than shape, using noise injection, flipping, cropping, resizing, and color-space manipulation.These methods expose models to varied image appearances during training while requiring preservation of the underlying information.
  • Image Manipulation: Flipping and cropping can preserve or corrupt labels depending on the dataset and the image region transformed.Horizontal flipping can turn an Urdu digit 2 into 6, while cropping can remove informative parts of an 8 and mislead the model.
  • Non-Geometric Data Augmentations: Noise injection can improve feature-learning robustness and defense against adversarial attacks, while color-space manipulation alters channels to reduce lighting-condition bias.Noise injection improved performance across nine UCI datasets; color-space augmentation changes channel values independently.
  • Non-Geometric Data Augmentations: Jitter changes brightness, contrast, saturation, and hue within specified ranges, whereas kernel filtering softens or sharpens images with Gaussian-blur or edge filters.Improper jitter ranges can damage content, and kernel filtering uses an n x n window.

2) Image Erasing Data Augmentations:

Image erasing and mixing augmentations diversify training views by removing, masking, or recombining image regions, while advanced methods also target local features, salient content, and style.

  • Image Erasing Data Augmentations: Image erasing removes image regions using methods including Cutout, Random Erasing, Hide-and-Seek, and GridMask.These methods differ in masking decisions, region structure, and replacement strategy.
  • Image Erasing Data Augmentations: Hide-and-Seek randomly removes uniform squares at each epoch to encourage networks to learn relevant features, but may remove important information.The method presents a different image view during each epoch.
  • Image Erasing Data Augmentations: GridMask applies a uniform masking pattern to balance the risk of erasing complete objects or removing contextual information.The method creates a mask and multiplies it with the input image.
  • Image Mixing Data Augmentations: Image mixing combines one or more images, including the same image, through single-image and non-single-image strategies.The survey compares these categories on benchmark datasets including CIFAR10, CIFAR100, and ImageNet.
  • Image Mixing Data Augmentations: LocalAugment divides an image into patches and applies different augmentations to increase local-feature diversity, though it does not preserve global structure.Self-Augmentation instead crops and pastes a random region, combining regional dropout with knowledge distillation for few-shot learning.
  • Image Mixing Data Augmentations: Object-focused and salient-region methods include ObjectAug, SalfMix, RICAP, and CutThumbnail, which manipulate object boundaries, salient regions, multiple image patches, or resized images.CutThumbnail keeps image content global in a small resized image, while ObjectAug applies different transformations to individual objects for semantic segmentation.
  • Image Mixing Data Augmentations: Mixup mixes image labels with the same factor used for images and improved accuracy and robustness across several datasets.The cited experiments include ImageNet-2012, CIFAR10, CIFAR-100, Google Commands 4, and UCI datasets.
  • Image Mixing Data Augmentations: StyleMix and StyleCutMix address the inability of mixup-based methods to distinguish image content from style features.The survey reports impressive performance on popular benchmark datasets for approaches that separately handle content and style.

2) AutoAugment:

AutoAugment-related methods search training data for augmentation policies rather than relying solely on manual selection. The surveyed approaches reduce search cost, extend policies to patches or detection boxes, and include reinforcement-learning and non-reinforcement-learning strategies.

  • AutoAugment: Automated augmentation searches training data for policies using a search algorithm and a defined search space.The survey divides these techniques into reinforcement-learning and non-reinforcement-learning categories.
  • AutoAugment: AutoAugment uses reinforcement learning to find sub-policies, each defined by an image-processing operation, probability, and magnitude.The method replaces manual augmentation selection with policy search.
  • AutoAugment: Fast AutoAugment reduces AutoAugment’s search time through density matching and reports lower-order training-time requirements than AutoAugment.Its overall procedure is described as an efficient search strategy.
  • AutoAugment: Faster AutoAugment uses differentiable policy search to estimate gradients for transformations with discrete parameters and select operations efficiently.The approach is presented as an acceleration of AutoAugment policy discovery.
  • AutoAugment: RAD uses traditional data augmentation to target learning efficiency and generalization in reinforcement-learning environments.The survey reports that these augmentations enable reinforcement-learning algorithms to outperform complex state-of-the-art tasks in pixel-based control and state-based settings.
  • AutoAugment: Multi-agent reinforcement learning searches for patch-level policies by dividing images into patches and jointly selecting augmentation policies for each patch.The method is described as producing competitive results on state-of-the-art benchmarks.
  • AutoAugment: Object-detection augmentation learns operations and values specifically for detection, addressing the mismatch between classification policies and bounding-box transformations.The surveyed work emphasizes that detection augmentation must be selected carefully because geometric changes add bounding-box complexity.

Non-Reinforcement Learning data augmentations:

Non-reinforcement-learning augmentation methods avoid reinforcement-learning search or use specialized strategies for object detection and low-data settings. These approaches include randomly parameterized policies, learned magnitude ranges, adversarial perturbations, feature-preserving dataset expansion, and perspective transformations.

  • Non-reinforcement-learning augmentation excludes methods that use reinforcement learning to find augmentation policies.The category includes approaches that do not require a reinforcement-learning algorithm for augmentation search.
  • RandAugment removes a separate search phase by randomly applying N augmentations with magnitude M.This reduces the complexity and computational overhead associated with searching for an augmentation policy.
  • RangeAugment learns augmentation magnitude ranges using an image-similarity auxiliary loss instead of manually defined ranges.The loss controls magnitude ranges to produce more effective policies.
  • Adversarial object-detection augmentation formulates policy selection through a game-theoretic Nash-equilibrium interpretation.It seeks perturbations of ground-truth boxes that force the predictor to learn from difficult sample distributions.
  • Deep CNN ensemble augmentation expands PASCAL VOC examples with consistent Microsoft COCO samples, improving performance.The method also uses several CNN models as an ensemble within an R-CNN variant.
  • Perspective transformation generates object-detection images from different viewing angles and showed effectiveness across several datasets.Deep adversarial augmentation separately targets extremely low-data regimes with class-conditional supervised GAN training.

4) Feature augmentation:

Feature augmentation transforms images into representations and applies augmentation in embedding space. Reviewed methods generate diverse features, address class imbalance, support domain adaptation, and complement image-space warping.

  • Feature augmentation performs transformations on image embeddings or representations rather than directly on images.The survey identifies it as a distinct augmentation category with a limited but growing body of work.
  • FeatMatch combines complex feature augmentations with consistency regularization for semi-supervised learning.It exploits intra-class and inter-class representations obtained through clustering.
  • FeatMatch produced an absolute 17.44% gain on miniImageNet and showed robustness on out-of-distribution samples.The reported gains were demonstrated for the proposed feature-based semi-supervised method.
  • Encoder-decoder feature augmentation applies noise, interpolation, and extrapolation to learned representations, improving static and sequential data.The interpolation and extrapolation process is illustrated for handwritten characters.
  • Long-tailed feature augmentation separates generic and class-specific features, then synthesizes under-represented samples from confusing classes.Class activation maps guide the separation and feature injection process.
  • Feature augmentation can be applied in image or representation space, including data warping and synthetic oversampling.The survey notes that general augmentation may transfer to feature space when suitable data augmentations are known.

5) Neural Style Transfer:

Neural style transfer augments training data by varying appearance while preserving high-level semantics. The surveyed methods randomize style, reduce dataset bias, and improve classification performance across several domains.

  • Style augmentation randomizes color, contrast, and texture while maintaining image shape and semantic content.It selects arbitrary style-transfer networks and target styles from a multivariate normal distribution embedding.
  • StyPath transfers style to reduce bias in kidney antibody-mediated-rejection classification and is faster than state-of-the-art augmentations.The survey illustrates the comparison between content-based and random initialization outcomes.
  • An artistic-style neural algorithm separates content and style embeddings before recombining them into artistic images.The resulting images are described as having high perceptual quality.
  • CycleGAN augmentation converts healthy-category COVID-19 X-rays into positive COVID images to balance and diversify the dataset.The method improved performance across several CNN architectures.

III. RESULTS

The survey compiles detailed results on how data augmentation affects image classification, object detection, and semantic segmentation. It also illustrates synthetic COVID-image generation, where image quality improves as training epochs increase.

  • The results section evaluates augmentation effects across image classification, object detection, and semantic segmentation.Results are compiled from multiple state-of-the-art data-augmentation studies.
  • Synthetic COVID images generated from the healthy category improve in quality as the number of training epochs increases.The figure presents the progression of generated-image quality during training.

A. Image Classification

This section presents results for several state-of-the-art data augmentation methods in supervised and semi-supervised learning.

  • The section compares state-of-the-art augmentation methods for supervised and semi-supervised learning.

1) supervised learning results:

The supervised-learning comparison evaluates state-of-the-art augmentation methods on CIFAR10, CIFAR100, and ImageNet using accuracy across several network families. The reported results show that each augmentation significantly improves accuracy.

  • The comparison uses CIFAR10, CIFAR100, and ImageNet with WideResNet, Pyramid Network, and ResNet variants.
  • Accuracy is the evaluation metric, with higher accuracy indicating better performance.
  • The tables distinguish state-of-the-art augmentations used alone from those combined with traditional flipping, rotation, and cropping.
  • Each evaluated data augmentation significantly improves accuracy in the reported comparisons.

2) Semi-supervised learning:

Semi-supervised learning uses limited labeled data alongside large-scale unlabeled data, and the survey compiles results for augmentation-based SSL methods. These methods increase data diversity and improve performance across several datasets and architectures.

  • Semi-supervised learning combines limited labeled data with large-scale unlabeled data.
  • Data augmentation is used with limited labeled data to increase training-data diversity.
  • Augmentation-based SSL methods improve performance across multiple datasets and neural-network architectures.
  • The surveyed SSL methods include pseudoLabel, SSL with memory, label propagation, and mean teacher.
  • The compiled datasets include CIFAR10, CIFAR100, SVHN, and Mini-ImageNet.

B. Object detection

The survey compiles augmentation results for object detection and semantic segmentation across commonly used VOC, COCO2017, and Cityscapes datasets. It reports gains on selected metrics, while highlighting unresolved questions about optimal augmentation and sample-generation choices.

  • B. Object detection: Object-detection results are compiled on COCO2017, PASCAL VOC, VOC 2007, and VOC 2012.
  • B. Object detection: FRCNN with synthetic data achieves the best mAP accuracy on VOC 2007 in the reported table.
  • B. Object detection: Semantic-segmentation results are compiled on PASCAL VOC and Cityscapes using validation-set performance.
  • B. Object detection: The segmentation evaluations report mean intersection over union (mIoU), while the survey also identifies gains in mAP.
  • B. Object detection: The survey states that performance improvement from additional augmentations reaches a limit after a certain extent.
  • B. Object detection: Combined augmentation methods show better performance than single augmentations.
  • B. Object detection: No theoretical support explains why particular augmentations improve performance or which samples should be augmented.
  • B. Object detection: Finding the optimal number of generated samples remains challenging because current approaches require many experiments and computational cost.

D. Selection of data augmentation based on model architecture and dataset

Data augmentation should be selected according to the dataset, model architecture, and task rather than applied uniformly. Open challenges include choosing the appropriate space, handling labels for erased content, targeting difficult examples, and optimizing augmentation order and quantity.

  • Geometric transformations can be unsafe when they change label information, such as rotating MNIST 6 and 9 digits.
  • Augmentation strength must match model capacity: weak augmentation can overfit densely parameterized CNNs, whereas excessive augmentation can harm shallow CNN generalization.
  • Choosing between data-space and feature-space augmentation remains an open challenge that depends on the dataset, model architecture, and task.
  • For image erasing methods such as Cutout, assigning mixed labels to reflect the lost image portion remains an unexplored research question.
  • Future work should assess augmentation on difficult examples and determine which samples, operations, order, and quantity to use.
  • The survey compiles techniques, task results, supervised and semi-supervised comparisons, available code, and unresolved challenges for reproducible evaluation.
Loading 2301.02830v4…