PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial AutoAugment

Xinyu Zhang, Qiang Wang, Jian Zhang, Zhao Zhong

arXiv:1912.11188v1cs.CVcs.LGstat.ML

TL;DR

AutoAugment-style policy search can be computationally impractical at scale and may transfer weakly across tasks. Adversarial AutoAugment jointly searches policies during target-network training, improving performance while reusing computation and avoiding retraining.

  • Problem

    Automatically learned augmentation policies improve generalization, but AutoAugment requires substantial computation and manually designed policies can transfer weakly across tasks and datasets.

  • Method

    Adversarial AutoAugment jointly optimizes target-network training and augmentation-policy search online, using adversarial policies to generate harder samples and reusing training computation for policy evaluation.

  • Results

    The method outperforms previous augmentation methods across CIFAR-10 and ImageNet; on ImageNet, ResNet-50 top-1 accuracy improves from 76.3% to 79.4% without extra data.

  • Takeaways & Limitations

    Adversarial policies promote robust feature learning, while online search and reused computation make augmentation-policy optimization more computing-efficient.

Abstract

from arXiv · show

Data augmentation (DA) has been widely utilized to improve generalization in training deep neural networks. Recently, human-designed data augmentation has been gradually replaced by automatically learned augmentation policy. Through finding the best policy in well-designed search space of data augmentation, AutoAugment can significantly improve validation accuracy on image classification tasks. However, this approach is not computationally practical for large-scale problems. In this paper, we develop an adversarial method to arrive at a computationally-affordable solution called Adversarial AutoAugment, which can simultaneously optimize target related object and augmentation policy search loss. The augmentation policy network attempts to increase the training loss of a target network through generating adversarial augmentation policies, while the target network can learn more robust features from harder examples to improve the generalization. In contrast to prior work, we reuse the computation in target network training for policy evaluation, and dispense with the retraining of the target network. Compared to AutoAugment, this leads to about 12x reduction in computing cost and 11x shortening in time overhead on ImageNet. We show experimental results of our approach on CIFAR-10/CIFAR-100, ImageNet, and demonstrate significant performance improvements over state-of-the-art. On CIFAR-10, we achieve a top-1 test error of 1.36%, which is the currently best performing single model. On ImageNet, we achieve a leading performance of top-1 accuracy 79.40% on ResNet-50 and 80.00% on ResNet-50-D without extra data.

1 INTRODUCTION

The paper proposes an efficient adversarial approach that jointly searches augmentation policies and trains a target network, avoiding the computational burden of prior policy search. It reports improved performance on CIFAR-10 and ImageNet while reducing search overhead.

  • AutoAugment improves image-classification accuracy but requires evaluating thousands of sampled policies, making its computing cost impractical for large-scale problems.
  • The method searches augmentation policies directly on the target task and full dataset during target-network training.
  • The augmentation policy network generates adversarial policies that increase target-network training loss, while harder samples promote robust feature learning.
  • 1.36% top-1 test error is achieved with PyramidNet+ShakeDrop on CIFAR-10, reported as state-of-the-art performance.
  • 79.4% top-1 accuracy is achieved with ResNet-50 on ImageNet without extra data, improving from 76.3% and exceeding AutoAugment by 1.77%.

2 RELATED WORK

Prior work established that augmentation can be learned automatically, while related adversarial generative methods provide a distinct comparison point. The paper situates its approach within automated policy learning and dataset enlargement.

  • Human-designed augmentation policies are dataset-specific and may transfer weakly across tasks, motivating automatic policy learning.
  • Smart Augmentation learns an augmentation network while training a target network by merging two or more samples from a class.
  • AutoAugment automatically learns augmentation policies through a designed policy search space and an RNN sample controller.
  • Unlike traditional GANs, the proposed method formulates augmentation-policy learning as a Min-Max game rather than directly synthesizing new images.

3 METHOD

Adversarial AutoAugment jointly trains a target network and an augmentation-policy controller, dynamically sampling harder policies while reusing target-network computation for efficient search.

  • Dynamic policies: The augmentation policy changes with the target network’s training state instead of remaining fixed throughout training.The policy network learns from the intermediate target-network state, producing more aggressive and adaptive policies.
  • Search space: Each policy contains 5 sub-policies, each with two ordered operations; the search space includes 16 image operations and omits operation probabilities.The omitted probability parameter is intended to avoid delaying feedback from the target network’s intermediate state.
  • Adversarial framework: The framework jointly optimizes target-network training and augmentation-policy search in an adversarial, online manner.The policy network increases target-network training loss, while the target network learns from harder augmented examples.
  • Target-network training: The target network trains on M augmented instances of each input, treating them as a larger N · M batch or averaged gradient computation.This formulation is intended to reduce gradient variance and accelerate convergence, although it can also increase overfitting.
  • Policy controller: The controller is an RNN that predicts 20 discrete parameters to form a complete policy, repeating the sub-policy architecture across Q sub-policies.The controller predicts operation types and magnitudes for the generated policy.
  • Policy optimization: REINFORCE optimizes the policy network because non-differentiable augmentation operations interrupt gradient flow from the target network.A moving-average and normalized mini-batch loss are used to reduce the variance of the policy-gradient estimate.

4 EXPERIMENTS AND ANALYSIS

Experiments across CIFAR-10, CIFAR-100, and ImageNet show that Adversarial AutoAugment improves performance while substantially reducing search overhead. Ablations and policy analyses indicate that dynamically generated adversarial policies support robust feature learning and adapt during training.

  • CIFAR-10 and CIFAR-100: M=8 balances performance and computing cost because accuracy improves rapidly up to M=8, while larger M values provide no significant improvement.The comparison evaluates M ∈{2, 4, 8, 16, 32} with Wide-ResNet-28-10 on CIFAR-10.
  • CIFAR-10 and CIFAR-100: The method achieves state-of-the-art performance on CIFAR-10 and CIFAR-100, including a 1.36% top-1 test error with PyramidNet+ShakeDrop on CIFAR-10.On CIFAR-10, it improves Wide-ResNet-28-10 over AutoAugment by 0.78% and over PBA by 0.68%.
  • CIFAR-10 and CIFAR-100: The learned policies increasingly select operations such as TranslateY, Rotate, Posterize, and SampleParing, while favoring more geometric transformations than color-focused AutoAugment.At the end of training, low magnitudes remain appreciable rather than being replaced entirely by extreme transformations.
  • ImageNet: 79.40% top-1 accuracy is achieved on ResNet-50 without extra data, and 80.00% on ResNet-50-D through architecture substitution alone.The paper describes the ResNet-50 result as the highest top-1 accuracy for ResNet-50 learned on ImageNet, to the authors’ knowledge.
  • Ablation study: In the ablation study, adversarial policies produce the best reported test error of 20.60%, outperforming fixed and randomly generated policies.Fixed reduces error by 0.99% versus Baseline, while Random is 1.02% better than Fixed.

5 CONCLUSION

The paper introduces adversarial learning for automatic data augmentation, jointly training the target network and searching policies online. Reusing network-training computation reduces search cost while adversarial policies encourage robust feature learning and improve performance.

  • Adversarial policies combat target-network overfitting by generating harder augmented samples during training.The target network responds by learning robust features, which leads to improved generalization.
  • The target network and augmentation policy search are optimized together in an online training process.Policy search proceeds alongside target-network training rather than as a separate retraining stage.
  • Reusing network-training computation for policy evaluation makes the augmentation search substantially more computing-efficient.The paper identifies computation reuse as the mechanism that reduces search cost.

A.1 HYPERPARAMETERS

The reported hyperparameters cover CIFAR-10, CIFAR-100, and ImageNet. Most settings follow previous work without specific tuning, with the number of epochs treated differently.

  • Model hyperparameters are detailed for CIFAR-10, CIFAR-100, and ImageNet in Table 7.
  • The learning rate and weight decay are represented by LR and WD, respectively.
  • The hyperparameters are not specifically tuned and remain consistent with previous works except for the number of epochs.
Loading 1912.11188v1…