PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Survey of Image Augmentation Techniques for Deep Learning

Mingle Xu, Sook Yoon, Alvaro Fuentes, Dong Sun Park

arXiv:2205.01491v2cs.CV

TL;DR

Training deep learning models requires large volumes of images, making image augmentation methods important to understand. The paper surveys these methods using a three-category taxonomy and reports task-specific benefits and empirical limitations.

  • Problem

    Training such a model requires a large volume of images, while understanding existing augmentation methods is crucial.

  • Method

    The paper presents a comprehensive survey with a novel taxonomy encompassing model-free, model-based, and optimizing policy-based augmentation categories.

  • Results

    Image augmentation significantly improves task performance, with methods designed for challenges including occlusion, class imbalance, and domain differences.

  • Takeaways & Limitations

    The survey supports selecting or developing augmentation algorithms for specific practical challenges.

  • Takeaways & Limitations

    Most image augmentation algorithms are designed for classification, and the evidence is empirical from experiments.

Abstract

from arXiv · show

Deep learning has been achieving decent performance in computer vision requiring a large volume of images, however, collecting images is expensive and difficult in many scenarios. To alleviate this issue, many image augmentation algorithms have been proposed as effective and efficient strategies. Understanding current algorithms is essential to find suitable methods or develop novel techniques for given tasks. In this paper, we perform a comprehensive survey on image augmentation for deep learning with a novel informative taxonomy. To get the basic idea why we need image augmentation, we introduce the challenges in computer vision tasks and vicinity distribution. Then, the algorithms are split into three categories; model-free, model-based, and optimizing policy-based. The model-free category employs image processing methods while the model-based method leverages trainable image generation models. In contrast, the optimizing policy-based approach aims to find the optimal operations or their combinations. Furthermore, we discuss the current trend of common applications with two more active topics, leveraging different ways to understand image augmentation, such as group and kernel theory, and deploying image augmentation for unsupervised learning. Based on the analysis, we believe that our survey gives a better understanding helpful to choose suitable methods or design novel algorithms for practical applications.

1. Introduction

The introduction motivates image augmentation as a response to costly, difficult data collection and limited datasets, then presents the survey’s broad scope, taxonomy, and related topics.

  • Deep-learning models require large image volumes, but collecting images is expensive and challenging, especially for practical applications with limited datasets.
  • The survey covers diverse augmentation algorithms, including generative adversarial networks, image mixing, instance-level multiple image mixing, and model-based methods.
  • Its application scope includes image classification, object detection, segmentation, tracking, and other image and video applications rather than one specific image type or task.
  • Image augmentation can mitigate computer-vision challenges, expand vicinity distribution, and improve trained-model generalizability.
  • The paper introduces challenges and vicinity distribution, presents a novel informative taxonomy, and discusses current directions including understanding augmentation, new strategies, and feature augmentation.

2. Taxonomy

The taxonomy divides image augmentation into model-free, model-based, and optimizing policy-based approaches according to how images or augmentation policies are produced.

  • Model-free augmentation uses no pre-trained model and may operate on single or multiple images.
  • Model-based augmentation generates images with trained models and may be unconditional, label-conditional, or image-conditional.
  • Optimizing policy-based methods select operations and suitable parameters from a large parameter space.
  • Optimizing policy-based methods are further divided into reinforcement-learning-based and adversarial-learning-based approaches.
  • Policy-search methods can use diverse operations, magnitudes, and an agent to find an optimal policy, whereas adversarial methods identify operations with corresponding magnitudes for the task model.

3. Motivation to perform image augmentation

Image augmentation addresses challenges including image variation, class imbalance, domain shift, and data remembering in deep learning. By increasing the number and variance of training images and moving samples within their vicinity, augmentation can improve model generalizability.

  • Challenges: Computer vision faces challenges from image variation, class imbalance, domain shift, and data remembering.These challenges arise from differing visual conditions, uneven class frequencies, differing training and testing distributions, and models memorizing limited training data.
  • Challenges: In automatic driving, daytime images may be easier to collect even when evaluation occurs at nighttime.This illustrates domain shift between training and testing conditions.
  • Challenges: Image augmentation can mitigate these challenges by increasing the number and variance of images in the training dataset.The stated purpose is to improve model generalizability and support suitable method selection for different applications.
  • Challenges: The survey provides an informative overview of image augmentation algorithms to improve understanding of methods for practical applications.Its stated contribution is a survey covering a wide range of image augmentation algorithms.
  • Vicinity distribution: Vicinity distributions replace point-mass empirical examples with local density around each data point.The vicinity distribution assumes that the underlying distribution is smooth around each point.
  • Vicinity distribution: Augmentation moves an original data point within its vicinity; Gaussian vicinity distribution corresponds to adding Gaussian noise.Models trained with vicinity distributions are described as less prone to memorizing all data points and tending to yield higher testing performance.

4. Model-free image augmentation

Model-free image augmentation uses image-processing operations to alter single images or combine multiple images without trainable generation models. The survey covers geometric, color, intensity, masking, and multiple-image methods, emphasizing label preservation, distribution changes, and practical trade-offs.

  • Overview: Model-free augmentation applies image-processing methods and divides into single-image and multiple-image branches.Single-image methods generate outputs from one image, whereas multiple-image methods generate outputs from multiple images.
  • Single-image augmentation: Single-image augmentation fluctuates training data points and increases distribution density using geometric, color, and intensity transformations.Geometric operations modify spatial relationships, color processing varies image colors, and intensity transformations change image parts.
  • Geometric transformation: Geometric transformations alter object position, perspective, orientation, or scale, but operation magnitude and direction must preserve label identity.For example, Cityscapes can be augmented horizontally but not vertically, and rotation angles require careful selection.
  • Color image processing: Color processing can yield superior performance but is rarely used when training and testing color variations are small.Color processing has also been used to obtain robust features for task-agnostic contrast learning.
  • Intensity transformation: Intensity transformations use pixel- or patch-level changes, with random noise providing a simple example and patch-level methods helping models learn robust features.The survey links these changes to avoiding trivial solutions.
  • Masking augmentation: Masking methods simulate occlusion: Hide-and-Seek blocks randomly selected equal-sized patches, Random Erasing randomizes area size, aspect ratio, and location, and GridMask balances deletion with reservation.GridMask varies grid dimensions and spacing, while its preliminary experiment reports fewer failure cases than Cutout and Hide-and-Seek.
  • Multiple-image augmentation: Multiple-image methods merge inputs through pixel averaging, label or image fusion, spatial composition, or instance-level reuse.Mixup combines images and one-hot labels using λ, CutMix assigns labels from pixel ratios, and Mosaic uses four images to increase object count.

5. Model-based image augmentation

Model-based augmentation uses pretrained image-generation models to create synthetic samples under unconditional, label-conditional, or image-conditional settings. These methods address class imbalance, domain shift, and robustness, but face limitations involving label control, fidelity, and training data balance.

  • Overview: Model-based augmentation requires a pretrained model to generate images and is classified as unconditional, label-conditional, or image-conditional.Image-conditional generation is further divided into label-preserving and label-changing forms.
  • Unconditional image generation: Unconditional models generate samples from noise and can approximate the original dataset distribution without reproducing its exact images.Interpolating latent vectors can produce intermediate images viewed as fluctuations between original data points.
  • Label-conditional image generation: Label-conditional generation supports class rebalancing by generating minority-class images, but imbalanced data can cause the generator to favor the majority class.Pretrained autoencoders with reconstruction loss have been used to initialize generators for this challenge.
  • Label-conditional image generation: One-hot label conditioning restricts trained models to known classes, motivating image-conditioned methods such as DAGAN that infer class information from an input image.DAGAN uses an image encoder, and the generated image is assumed to retain the original image’s class.
  • Image-conditional image generation: Image-conditioned translation changes style while preserving or changing content, supporting domain adaptation, robustness, class-imbalance mitigation, and semantic transformations.Style translation can encourage CNNs to attend more to shape, while label-changing methods alter label-dependent content and style simultaneously.
  • Image-conditional image generation: Label-changing translation yields more significant robustness improvements than label-preserving translation by changing label and style simultaneously.Generated images may have poor fidelity, although the approach improves emotion classification.

6. Optimizing policy-based image augmentation

Policy-based augmentation searches for operations, probabilities, magnitudes, or schedules that optimize task performance instead of manually fixing augmentation settings. Reinforcement-learning and adversarial-learning approaches reduce manual tuning but introduce computational, task-transfer, and policy-granularity limitations.

  • Overview: Policy-based optimization addresses the challenging, time-consuming search for dataset-specific augmentation operations and magnitudes.Its two categories are reinforcement-learning-based and adversarial-learning-based augmentation.
  • Reinforcement-learning-based augmentation: Reinforcement-learning methods sample augmentation strategies and use validation accuracy to update a controller.The controller selects operation type, probability, and magnitude, while lighter child networks reduce the cost of searching a large space.
  • Reinforcement-learning-based augmentation: AutoAugment achieves satisfactory classification performance across several datasets but requires long training and may learn a policy that is suboptimal for the final task model.RandAugment, Fast AA, Faster AA, and PBA address search cost or policy-learning design from different perspectives.
  • Policy granularity: Later methods refine policy granularity by learning class-level, sample-level, or instance-level policies rather than one dataset-level policy.Instance-level augmentation has been considered for object detection, with operations performed inside bounding boxes.
  • Adversarial-learning-based augmentation: Adversarial augmentation learns policies that generate hard samples by increasing training loss while maintaining the original class constraints.Some methods select samples for augmentation when the augmented image produces larger training loss than the original.
  • Testing-stage augmentation: AdaTransform extends augmentation to testing by increasing input variance during competitive training and reducing variance during cooperative testing.The transformer, discriminator, and target network jointly support these two phases.

7. Discussions

The discussion highlights strong empirical benefits from augmentation while identifying open questions about theory, application scope, feature-space methods, and unsupervised use. It also points toward system-level analysis and application-specific augmentation design.

  • Current situation: Image augmentation produces impressive results across multiple datasets and heterogeneous models, including reported gains on ImageNet-2012 and 102Flowers.Reported examples include validation-accuracy increases of 1.5 and 1.2 percent for ResNet-50 and ResNet-101, respectively.
  • Current situation: Model-free methods and optimizing policies are widely used, while model-based augmentation remains an active topic for class imbalance and domain shift.Label-changing methods have also recently received attention alongside predominantly label-preserving algorithms.
  • Future directions: Future work should extend augmentation beyond classification, incorporate application-specific knowledge, and analyze its effects on optimization schedules such as learning rate and batch size.The discussion also proposes distinguishing specialized applications from general benchmarks when designing new methods.
  • Understanding image augmentation: Theoretical understanding remains limited because evidence that augmentation resembles regularization is mostly empirical.Kernel theory and group theory have been used to analyze augmentation effects, while affinity and diversity quantify generalization-related improvement.
  • New strategy to leverage image augmentation: Augmentation can support unsupervised and semi-supervised learning through pretext tasks, contrastive positive samples, and semi-supervised training.Examples include predicting rotation degrees or relative patch positions and treating augmented images as similar to originals.
  • Feature augmentation: Feature augmentation performs transformations in feature space, reducing computation cost and potentially exploiting denser semantic information than image space.The survey suggests extending image-augmentation techniques such as Mixup to feature augmentation.

8. Conclusion

The survey organizes image augmentation into model-free, model-based, and optimizing policy-based categories and relates methods to computer-vision challenges. It concludes that augmentation improves task performance while motivating challenge-specific methods and broader applications.

  • Conclusion: The survey proposes a taxonomy with three categories: model-free, model-based, and optimizing policy-based augmentation.It also analyzes augmentation objectives through computer-vision challenges and vicinity distribution.
  • Conclusion: Image augmentation significantly improves task performance, with methods designed for challenges such as occlusion, class imbalance, and domain shift.The conclusion associates intensity transformations with occlusion and model-based algorithms with class imbalance and domain shift.
  • Conclusion: Recognizing new challenges or variations may inspire novel augmentation methods when those conditions can be mimicked through augmentation.The survey presents challenge recognition as a basis for developing new algorithms.
  • Conclusion: The survey aims to help practitioners select methods after recognizing dataset-specific challenges and to encourage prioritizing dataset characteristics in augmentation design.It also discusses current situations and possible directions through three related topics.
Loading 2205.01491v2…