Source-linked AI summary
Faster AutoAugment: Learning Augmentation Strategies using Backpropagation
Ryuichiro Hataya, Jan Zdenek, Kazuki Yoshizoe, Hideki Nakayama
TL;DR
Existing augmentation-policy searches operate over large spaces and often rely on time-consuming black-box optimization. Faster AutoAugment enables differentiable policy search through gradient approximations, differentiable operation selection, and distribution matching. It achieves much faster policy searching with competitive performance on standard benchmarks, while the authors identify future applications rather than a demonstrated limitation.
Problem
Data augmentation policy search spans prohibitively large spaces, while discrete parameters and non-differentiable operation selection make gradient-based optimization difficult.
Method
Faster AutoAugment approximates gradients for discrete image operations, differentiates operation selection, and minimizes the distribution distance between original and augmented images using adversarial learning.
Results
Faster AutoAugment achieves significantly faster policy searching than prior methods without a performance drop and comparable performance on standard benchmarks.
Takeaways & Limitations
End-to-end differentiable policy optimization provides a faster alternative to prior automatic augmentation searches while retaining competitive benchmark performance.
Takeaways & Limitations
The authors describe learning from small data and applications to representation learning as future directions rather than demonstrated results.
Abstract
from arXiv · showhide
Data augmentation methods are indispensable heuristics to boost the performance of deep neural networks, especially in image recognition tasks. Recently, several studies have shown that augmentation strategies found by search algorithms outperform hand-made strategies. Such methods employ black-box search algorithms over image transformations with continuous or discrete parameters and require a long time to obtain better strategies. In this paper, we propose a differentiable policy search pipeline for data augmentation, which is much faster than previous methods. We introduce approximate gradients for several transformation operations with discrete parameters as well as the differentiable mechanism for selecting operations. As the objective of training, we minimize the distance between the distributions of augmented data and the original data, which can be differentiated. We show that our method, Faster AutoAugment, achieves significantly faster searching than prior work without a performance drop.
1. Introduction
Data augmentation strategy design is a large, difficult search problem because it combines operation choices and transformation magnitudes. Faster AutoAugment makes policy search end-to-end differentiable and reports substantially faster searching without a performance drop.
- Data augmentation increases the virtual amount and diversity of training data, especially improving performance in image recognition tasks.
- Designing augmentation strategies is a complex combinatorial problem involving operation selection, operation combinations, and magnitude settings.
- Gradient information is difficult to obtain because magnitude parameters may be discrete and operation selection is non-differentiable.
- Faster AutoAugment approximates gradients for non-differentiable operations and makes operation selection differentiable, enabling end-to-end gradient-based policy optimization.
- Faster AutoAugment significantly reduces search time compared with prior methods without a performance drop and achieves comparable performance on standard benchmarks.
2. Related Work
Automated data augmentation searches for effective combinations of image transformations, using black-box optimization, density matching, or generative models across recognition and representation-learning settings.
- Automating Data Augmentation: Automated augmentation searches for better combinations of symbolic image operations rather than relying solely on manually selected transformations.Search methods include reinforcement learning, evolution strategies, Bayesian optimization, and population-based training.
- Automating Data Augmentation: Black-box approaches optimize augmentation policies through reinforcement learning, evolution strategies, Bayesian optimization, or population-based training.Different methods either target classifier accuracy or match augmented and original image densities.
- Automating Data Augmentation: Generative-adversarial approaches create or modify images to improve classifier performance or make simulated objects resemble real ones.The cited methods use conditional GANs for image generation and simulator-output modification.
- Automating Data Augmentation: A policy randomly selects a sub-policy that applies consecutive image transformations, such as shear x and solarize, with operation-specific probabilities and magnitudes.
- Automating Data Augmentation: Automated augmentation also supports representation-learning applications including semi-supervised learning and domain generalization.
3. Preliminaries
The augmentation policy consists of randomly selected sub-policies containing consecutive operations with probability and magnitude parameters. Searching jointly over operations and parameters creates a very large space that prior black-box methods explore through repeated model evaluations.
- 3. Preliminaries: AutoAugment, PBA, Fast AutoAugment, and Faster AutoAugment use policies composed of randomly selected sub-policies applied to input images.
- 3. Preliminaries: Each sub-policy applies K consecutive image-processing operations one by one, where K is the operation count.
- 3.1. Operations: Operations include affine transformations, color enhancement, cutout, and sample pairing, with some magnitude parameters discrete, continuous, or absent.
- 3.1. Operations: Each operation uses a magnitude µ_O and application probability p_O, representing transformed output when applied and the original image otherwise.
- 3.1. Operations: A sub-policy composes operations as S(X; µ_S, p_S) = (O_K ◦ · · · ◦ O_1)(X; µ_S, p_S), with parameter vectors collecting operation magnitudes and probabilities.
- 3.2. Search Space: The search phase finds operation combinations and probability-magnitude parameter sets for L sub-policies, after which the policy augments data for neural-network training.
- 3.2. Search Space: With L = 10 and K = 2, discretizing 16 operations, 10 magnitudes, and 11 probabilities yields a search space of approximately 8.1 × 10^64.Prior black-box methods repeatedly train CNNs with candidate policies and obtain validation accuracy, whereas Faster AutoAugment uses gradient-based optimization to avoid these repetitive evaluations.
4. Faster AutoAugment
Faster AutoAugment turns policy search into gradient-based optimization by making image operations and operation selection differentiable, while matching augmented and original image distributions.
- Differentiable policy search: Faster AutoAugment explores augmentation policies with gradient-based optimization rather than brute-force or black-box search.The search targets operation combinations and parameter sets across multiple sub-policies, whose space grows prohibitively large for naive search.
- Differentiable policy search: Relaxed Bernoulli reparameterization makes each operation differentiable with respect to its probability parameter p.At low temperature, the relaxed distribution behaves like a Bernoulli distribution.
- Differentiable policy search: A straight-through estimator provides approximate gradients for operations whose magnitude parameters are discretized.The forward computation applies the exact augmentation, while the backward computation uses an approximated gradient.
- Differentiable policy search: Operation selection is made differentiable by replacing a selected operation with a softmax-weighted sum of all candidate-operation outputs during searching.After searching, operations are sampled according to the trained categorical weights.
- Density-matching objective: The learning objective minimizes the Wasserstein distance between original and augmented image distributions using a critic trained with Wasserstein GAN gradient penalty.Training also updates the sub-policy, operation, and classifier-related parameters using stochastic gradient descent and classification loss.
5. Experiments and Results
Experiments on CIFAR-10, CIFAR-100, SVHN, and ImageNet show that Faster AutoAugment achieves competitive performance with prior methods, including in low-resource settings, while searching significantly faster.
- Experimental setup: Experiments compare Faster AutoAugment with AutoAugment, PBA, and Fast AutoAugment across CIFAR-10, CIFAR-100, SVHN, and ImageNet.Except for ImageNet, experiments are run three times and averaged.
- CIFAR results: Faster AutoAugment achieves competitive test error rates with prior work on CIFAR-10 and CIFAR-100 across multiple CNN architectures.For CIFAR-100, policies trained on reduced CIFAR-100 outperform those trained on reduced CIFAR-10, suggesting the importance of using the target dataset.
- Augmented images: The learned policy appears to prefer color-enhancing operations for CIFAR-10, consistent with observations reported for AutoAugment.Examples of the resulting augmented images are shown in Figure 5.
- Low-resource setting: 7.7% improvement over Cutout is reported on Reduced CIFAR-10, with an error rate close to AutoAugment in the 4,000-image setting.The policy is trained and evaluated on the same randomly sampled subset.
- SVHN results: 1.2% error on SVHN is reported for Faster AutoAugment, a 0.1% improvement over Cutout and on par with PBA.The policy selects more geometric transformations on SVHN than on CIFAR-10.
- ImageNet results: Faster AutoAugment achieves a 1.0% improvement over the ImageNet baseline on top-1 error, close to the gains of AutoAugment and Fast AutoAugment.The authors note that they could not reproduce the prior reported baseline performance.
6. Analysis
Analysis examines how policy capacity and training-data size affect performance, finding benefits from more sub-policies and operations but a data-size-related degradation under a fixed policy capacity.
- Changing the Number of Sub-policies: More sub-policies lower CIFAR-10 test error, while increasing the number of sub-policies causes exponential growth in the search space.The number of sub-policies determines the diversity of augmented images.
- Changing the Operation Count: Increasing each sub-policy’s operation count from 1 to 4 improves performance, while also exponentially increasing the search space.The authors conclude that Faster AutoAugment scales to a large search space.
- Effect of Training Data Size: Increasing policy-training data from 4,000 to 50,000 CIFAR-10 images worsens error from 3.7% to 4.1% when using L = 10 sub-policies.The authors hypothesize that the degradation reflects insufficient policy capacity at L = 10.
- Effect of Policy Training: 2.6% error is achieved by trained policies versus 2.7% for randomly initialized policies on CIFAR-10 with WideResNet-28-10.Both error rates are averages over three runs.
7. Conclusion
Faster AutoAugment accelerates data-augmentation policy search through gradient approximation and end-to-end differentiability while retaining competitive benchmark performance. The authors identify faster search as potentially useful for representation learning and learnable policies for small-data settings.
- Conclusion: Faster AutoAugment uses gradient approximation for non-differentiable image operations to make policy search end-to-end differentiable.The method is evaluated on several standard benchmarks and achieves competitive performance with other automatic augmentation methods.
- Conclusion: Faster policy searching may benefit representation-learning research, including semi-supervised learning and domain generalization.The authors also identify learning from small data with learnable policies as an interesting future direction.