PaperlayerView exact revision on arXiv

Source-linked AI summary

RandAugment: Practical automated data augmentation with a reduced search space

Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, Quoc V. Le

arXiv:1909.13719v2cs.CV

TL;DR

Learned augmentation policies can improve performance but often require a costly separate search and may not transfer optimally across model and dataset sizes. RandAugment reduces the search space so augmentation can be tuned directly on the target task, matching or surpassing prior approaches across classification and detection benchmarks, including 85.0% ImageNet accuracy.

  • Problem

    Learned augmentation methods require costly separate optimization procedures and may not tailor regularization to the target model and dataset sizes.

  • Method

    RandAugment uses a dramatically reduced, two-hyperparameter search space that enables simple grid search directly on the target task without a separate proxy-task search.

  • Results

    RandAugment matches or surpasses prior approaches across CIFAR-10/100, SVHN, ImageNet, and COCO, achieving 85.0% ImageNet accuracy, 0.6% above previous methods and 1.0% above baseline augmentation.

  • Takeaways & Limitations

    RandAugment scales to large datasets such as ImageNet and COCO with minimal computational cost while maintaining notable predictive performance gains.

  • Takeaways & Limitations

    The method’s effects on robustness, semi-supervised learning, and other machine-learning domains remain open questions for future work.

Abstract

from arXiv · show

Recent work has shown that data augmentation has the potential to significantly improve the generalization of deep learning models. Recently, automated augmentation strategies have led to state-of-the-art results in image classification and object detection. While these strategies were optimized for improving validation accuracy, they also led to state-of-the-art results in semi-supervised learning and improved robustness to common corruptions of images. An obstacle to a large-scale adoption of these methods is a separate search phase which increases the training complexity and may substantially increase the computational cost. Additionally, due to the separate search phase, these approaches are unable to adjust the regularization strength based on model or dataset size. Automated augmentation policies are often found by training small models on small datasets and subsequently applied to train larger models. In this work, we remove both of these obstacles. RandAugment has a significantly reduced search space which allows it to be trained on the target task with no need for a separate proxy task. Furthermore, due to the parameterization, the regularization strength may be tailored to different model and dataset sizes. RandAugment can be used uniformly across different tasks and datasets and works out of the box, matching or surpassing all previous automated augmentation approaches on CIFAR-10/100, SVHN, and ImageNet. On the ImageNet dataset we achieve 85.0% accuracy, a 0.6% increase over the previous state-of-the-art and 1.0% increase over baseline augmentation. On object detection, RandAugment leads to 1.0-1.3% improvement over baseline augmentation, and is within 0.3% mAP of AutoAugment on COCO. Finally, due to its interpretable hyperparameter, RandAugment may be used to investigate the role of data augmentation with varying model and dataset size. Code is available online.

1. Introduction

RandAugment addresses the cost and transferability limitations of learned augmentation policies by reducing the search space to two interpretable hyperparameters and eliminating a separate proxy-task search. It achieves state-of-the-art results across image benchmarks and approaches state-of-the-art object detection performance.

  • Motivation: Manual augmentation policy design requires domain expertise, limiting portability across applications and domains.Data augmentation is used in image classification, object detection, instance segmentation, and speech recognition.
  • Motivation: Learned augmentation policies can improve accuracy, robustness, and semi-supervised learning without adding inference-time computational cost.The inference-time cost comparison is made with engineering larger network architectures.
  • Problem: Separate searches on small proxy tasks are computationally costly and may produce sub-optimal policies because augmentation strength depends on model and training-set size.The proxy-task formulation assumes that performance on the smaller task predicts performance on the larger target task.
  • Method: RandAugment removes separate search by using a dramatically reduced search space with 2 interpretable hyperparameters, making simple grid search sufficient to tailor policies to each model and dataset.The reduced search space allows policy selection directly on the target task rather than through a separate proxy task.
  • Results: 85.0% accuracy on ImageNet is 0.6% above previous methods and 1.0% above baseline augmentation, while object detection is within 0.3% mAP of state-of-the-art.RandAugment also achieves state-of-the-art results on CIFAR and SVHN.

2. Related Work

Prior work ranges from standard image transformations to learned methods that combine or search over augmentation operations. RandAugment builds on findings from AutoAugment, Population Based Augmentation, and Fast AutoAugment while simplifying how augmentation policies are selected and scheduled.

  • Individual augmentation operations: Common augmentation methods include horizontal flips and random crops or translations for natural images, and elastic distortions for MNIST.These operations are used in classification and detection models, while MNIST distortions vary scale, position, and orientation.
  • Learned augmentation strategies: Learned augmentation methods combine operations by merging same-class samples, modeling training-set distributions, or applying transformations in learned feature space.Examples include Smart Augmentation, Tran et al.’s Bayesian approach, and DeVries et al.’s transformations.
  • AutoAugment: AutoAugment uses reinforcement learning to select operation sequences, application probabilities, and magnitudes, with stochastic choices at several levels.Its stochasticity includes sub-policy selection, operation application, and direction for some operations.
  • Influences on RandAugment: Population Based Augmentation found that optimal augmentation magnitude increases during training, motivating RandAugment’s fixed magnitude schedule rather than transformation-specific magnitude searches.The schedule is discussed in detail in Section 3.

3. Methods

RandAugment folds augmentation choices into model-training hyperparameters, eliminating a costly separate proxy-task search phase. It reduces the policy space by uniformly selecting transformations and using a single global distortion magnitude, leaving two interpretable parameters, N and M.

  • RandAugment targets eliminating the separate proxy-task search phase because it complicates training, increases computation, and may produce sub-optimal results.
  • The method reduces learned policy complexity by replacing transformation-specific choices and probabilities over K=14 transformations with uniform transformation selection.
  • RandAugment uses a shared global distortion magnitude M because learned transformation magnitudes follow similar training schedules.Each transformation uses an integer strength scale from 0 to 10, with 10 representing its maximum scale.
  • The resulting algorithm has two human-interpretable parameters, N and M, and larger values increase regularization strength.Naive grid search is effective because the search space is extremely small.

4. Results

RandAugment is evaluated across image classification and object detection, where it matches or surpasses prior learned augmentation methods while using a shared transformation list. Results also show that augmentation strength depends on model and dataset size, motivating task-specific tuning without a proxy task.

  • Experimental scope: RandAugment is evaluated on CIFAR-10, CIFAR-100, SVHN, ImageNet, and COCO object detection against previous learned augmentation methods.The experiments replicate corresponding architectures and data transformations to measure relative benefits.
  • Model and dataset size: Optimal distortion magnitude increases monotonically with Wide-ResNet model size and CIFAR-10 training-set size.The reported trends persist despite measurement noise, indicating that augmentation strength should adapt to model and dataset scale.
  • Model and dataset size: A small proxy task may select augmentation strength tailored to the proxy rather than the larger task, motivating joint optimization of model weights and augmentation policy.The passage identifies proxy-task dependence as a potential source of sub-optimal policies.
  • SVHN: RandAugment matches or outperforms previous methods on SVHN with and without extra data, using the same transformation list without alteration.The comparison uses Wide-ResNet-28-2 and Wide-ResNet-28-10.
  • ImageNet: 85.0% accuracy on EfficientNet-B7 establishes a new ImageNet state of the art, with 1.0% improvement over baseline augmentation.On larger ImageNet models, RandAugment increases performance by up to +1.3% above baseline and matches AutoAugment and Fast AutoAugment on ResNet-50.
  • Transformation sensitivity: More than 1% average validation-accuracy improvement occurs with only two transformations, while median accuracy improves as the number of transformations increases.The analysis considers random subsets of 14 transformations and excludes flips, pad-and-crop, and cutout.

5. Discussion

The discussion identifies scaling and size-insensitive regularization as systematic drawbacks of prior learned augmentation methods. RandAugment scales to ImageNet and COCO with minimal computational cost and notable predictive gains, while a differentiable virtual-step approach produced consistently negative results with a second-order approximation.

  • Prior learned augmentation methods did not tailor the number of distortions or distortion magnitude to dataset and model size.
  • RandAugment scales to ImageNet and COCO with minimal computational cost, using 2 hyperparameters while delivering notable predictive performance gains.
  • Consistently negative results followed from learning αij by differentiating through a virtual training step with a 2nd-order approximation.The results are reported in Appendix A.1.

A. Appendix · A.1. Second order term from bilevel optimization

The appendix formulates augmentation-parameter optimization as a bilevel problem and derives a second-order approximation using a virtual training step. It also contrasts this with the first-order approximation and describes a finite-difference computation for the remaining term.

  • A.1. Second order term from bilevel optimization: Augmentation parameters α and network weights w are optimized jointly through a bilevel optimization problem.The objective is to find augmentation parameters whose training-set optimization produces strong validation performance.
  • A.1. Second order term from bilevel optimization: The bilevel problem is approximated with a single virtual training step.The approximation follows the formulation summarized from [30].
  • A.1. Second order term from bilevel optimization: The virtual update uses ξ as the virtual learning rate and expands the resulting validation-gradient expression.The expansion separates terms associated with the virtual update.
  • A.1. Second order term from bilevel optimization: When ξ = 0, the second term vanishes and the first term becomes ∇Lval(w, α), yielding the first-order approximation.This approximation was called the first-order approximation in [30].
  • A.1. Second order term from bilevel optimization: 0.3% out of 0.5% of architecture-search improvement was achieved with the first-order approximation.The approximation reduced the reported search time from 4 days to 1.5 days.
  • A.1. Second order term from bilevel optimization: The appendix notes that the first-order approximation behaves differently when α represents augmentation parameters.The supplied passage begins this qualification but does not include its continuation.
  • A.1. Second order term from bilevel optimization: The remaining computation uses w± = w ± ϵ∇w′Lval(w′, α), where ϵ is a small number.The supplied passage defines w± through a finite-difference-style perturbation around w′.

A.1.1 Magnitude methods

RandAugment considers several ways to set distortion magnitude, including random, constant, linearly increasing, and random-with-increasing-upper-bound schedules. Preliminary experiments found them equally effective, so the study uses constant magnitude because it has only one hyperparameter.

  • Magnitude can be sampled uniformly between two values or fixed to a constant number.
  • Other schedules linearly interpolate magnitude during training or increase a random magnitude’s upper bound linearly.
  • All magnitude strategies worked equally well in preliminary experiments, leading the study to select constant magnitude for the remaining experiments.Constant magnitude was chosen because it uses only a single hyperparameter.

A.1.2 Optimizing individual transformation magnitudes

Changing the magnitude of an individual transformation has only a minor effect on accuracy, supporting a shared magnitude M across transformations. The tied magnitude closely matches the optimal per-transformation magnitude.

  • Optimizing individual transformation magnitudes: A shared magnitude M causes only minor accuracy differences compared with optimizing each transformation magnitude separately.Across the reported settings, tied versus optimal magnitudes differ by 0.19% and 0.18% for rotation, and 0.07% and 0.05% for TranslateX.

A.2. Experimental Details … A.3. COCO

The experiments used task-specific training schedules and RandAugment hyperparameter searches across CIFAR, SVHN, ImageNet, and COCO. The number of operations and distortion magnitude were tuned separately for different datasets and model architectures.

  • A.2.1 CIFAR: Wide-ResNet models on CIFAR were trained for 200 epochs, while Shake-Shake and ShakeDrop models were trained for 1800 epochs with cosine learning-rate decay.The models used batch size 128, except ShakeDrop, which used batch size 64 because 128 did not fit on one GPU.
  • A.2.1 CIFAR: On CIFAR-10, RandAugment used N = 3 and magnitude values 4, 5, 7, 9, and 11 were tested, with optimal magnitudes varying by model.The optimal magnitudes were 4 for Wide-ResNet-2, 5 for Wide-ResNet-10, 9 for Shake-Shake, and 7 for PyramidNet + ShakeDrop.
  • A.2.2 SVHN: On core SVHN, cutout followed RandAugment, N = 3 was used, magnitudes 5, 7, 9, and 11 were tested, and magnitude 9 was optimal for both Wide-ResNet-28-2 and Wide-ResNet-28-10.Both models used a learning rate of 5e-3, weight decay of 5e-3, cosine decay, and 200 epochs.
  • A.2.2 SVHN: On full SVHN, N = 3 was used with magnitudes 5, 7, 9, and 11 tested; the optimal magnitude was 5 for Wide-ResNet-28-2 and 7 for Wide-ResNet-28-10.Training used 160 epochs, learning rate 5e-3, weight decay 1e-3, and cosine decay.
  • A.2.3 ImageNet: ImageNet ResNet models were trained for 180 epochs with standard ResNet-50 hyperparameters, global batch size 4096 across 32 workers, and magnitude 9 with N = 2 for ResNet-50.The image size was 224 by 244, weight decay was 0.0001, momentum was 0.9, and the learning rate was 0.1 scaled by batch size divided by 256.
  • A.2.3 ImageNet: ImageNet used horizontal flipping and random crops before RandAugment, with standard training and validation splits for training and evaluation.EfficientNet experiments used 350 epochs and a batch size of 4096 split across 256 replicas.
  • A.3. COCO: COCO training added horizontal flipping and scale jitters to RandAugment, used the classification transformation list, and transformed bounding boxes for geometric operations.The setup used learning rate 0.08, weight decay 1e 4, focal-loss parameters α = 0.25 and γ = 1.5, N = 1, and distortion magnitudes between 4 and 9.
Loading 1909.13719v2…