Source-linked AI summary
Fast AutoAugment
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, Sungwoong Kim
TL;DR
Existing augmentation policy search can improve image recognition but AutoAugment requires substantial computation. Fast AutoAugment uses density matching for efficient policy search and achieves much faster search with comparable performance across datasets and networks, including a 19.4% ImageNet error rate versus AutoAugment’s 20.0%.
Problem
AutoAugment improves image recognition but its policy search requires thousands of GPU hours even for relatively small datasets.
Method
Fast AutoAugment learns augmentation policies using an efficient density matching search that avoids repeated child-model training and back-propagation for each policy evaluation.
Results
Fast AutoAugment searches significantly faster than AutoAugment while retaining comparable performance across diverse datasets and networks; on ImageNet, ResNet-200 reaches 19.4% error versus AutoAugment’s 20.0%.
Takeaways & Limitations
The method supports direct augmentation search and transfer of learned policies to new datasets while substantially reducing search computation.
Takeaways & Limitations
Transferred policies have larger performance gaps as model capacity increases because policies searched with a small model have limited ability to improve larger models.
Abstract
from arXiv · showhide
Data augmentation is an essential technique for improving generalization ability of deep learning models. Recently, AutoAugment has been proposed as an algorithm to automatically search for augmentation policies from a dataset and has significantly enhanced performances on many image recognition tasks. However, its search method requires thousands of GPU hours even for a relatively small dataset. In this paper, we propose an algorithm called Fast AutoAugment that finds effective augmentation policies via a more efficient search strategy based on density matching. In comparison to AutoAugment, the proposed algorithm speeds up the search time by orders of magnitude while achieves comparable performances on image recognition tasks with various models and datasets including CIFAR-10, CIFAR-100, SVHN, and ImageNet.
1 Introduction
Deep learning models can overfit without abundant labeled data, making carefully designed augmentation important but traditionally dependent on human expertise. Fast AutoAugment addresses AutoAugment’s expensive search with density matching, achieving substantially faster search and comparable performance across datasets and networks.
- Large-capacity deep learning models often overfit without sufficiently large labeled datasets.
- Carefully designed augmentation improves generalization more than naive random transformations, but usually requires dataset-specific human expertise.
- Fast AutoAugment searches augmentation policies through density matching without repeated child-model training or back-propagation during each policy evaluation.
- Fast AutoAugment searches significantly faster than AutoAugment while retaining comparable performance across diverse image datasets and networks.
- 19.4% error on ImageNet with ResNet-200 is 0.6% better than AutoAugment’s 20.0% error.
2 Related Work
Image augmentation methods range from manually designed transformations to generative approaches and automated searches over predefined transformations. AutoAugment uses reinforcement learning, while PBA uses population-based training and Fast AutoAugment uses hyperparameter optimization.
- Manual baselines include random crop, flip, rotation, scaling, and color transformation, while Mixup, Cutout, and CutMix replace or mask image patches.
- Generative augmentation methods learn to create data by merging same-class samples, using GANs, or treating augmented data as missing points.
- Automated augmentation search methods select combinations of predefined transformations, unlike prior methods that directly generate augmented data with generative models.
- Each sub-policy contains two operations; an operation has an application probability and magnitude, and the sequence can produce an identity mapping.
- AutoAugment alternates training a child model and an RNN controller with reinforcement learning to search policies.
3 Fast AutoAugment
Fast AutoAugment searches augmentation policies by matching data densities and evaluating candidates with a single trained model, avoiding repeated child-model training. Its distributed, Bayesian-optimization-based procedure supports continuous policy parameters and parallel search before final retraining.
- Search Space: Each augmentation operation has a calling probability p and, when applicable, a magnitude λ; sub-policies compose such operations sequentially.The search space uses continuous probability and magnitude values in [0, 1].
- Policy Selection: The algorithm selects the top-N policies from each split, merges them into T∗, augments the full training set, and retrains the model.The resulting policy set is intended to improve performance on augmented data.
- Efficient Density Matching: Fast AutoAugment formulates augmentation-policy search as density matching between training data and augmented data, evaluated through model predictions.The model is trained on one split, while another split is used to explore policies.
- Search Procedure: The procedure splits training data into K stratified folds, trains model parameters on one split, and explores policies on another.The implementation trains models across splits and can parallelize this stage across machines.
- Policy Exploration: Bayesian optimization evaluates B candidate policies by tuning operation probabilities and magnitudes to minimize expected loss without retraining model parameters from scratch.Expected Improvement and variable kernel density estimation guide exploration in the graph-structured search space.
- Implementation: Fast AutoAugment is implemented with distributed learning and Ray, enabling parallel training and policy search for greater efficiency.The procedure combines split-wise training with HyperOpt-based exploration.
4 Experiments and Results
Fast AutoAugment is evaluated across CIFAR-10, CIFAR-100, SVHN, and ImageNet against baseline augmentation methods, Cutout, AutoAugment, and PBA. It generally improves over weaker baselines while reducing search cost and retaining comparable or better performance than AutoAugment.
- Experimental Setup: Fast AutoAugment is evaluated on CIFAR-10, CIFAR-100, SVHN, and ImageNet across multiple network architectures.The comparisons include baseline preprocessing, Cutout, AutoAugment, and PBA where applicable.
- CIFAR-10 and CIFAR-100 Results: Fast AutoAugment significantly improves baseline and Cutout performance on CIFAR-10 and CIFAR-100 while achieving comparable performance to AutoAugment.This pattern is reported for both direct search and policy transfer experiments.
- CIFAR-10 Results: 3.5 GPU-hours suffice for policy search on reduced CIFAR-10, while estimated full direct search requires 780 GPU-hours versus 5000 for AutoAugment.The reported 780 GPU-hour estimate is for Pyramid-Net+ShakeDrop in the worst case.
- CIFAR-100 Results: On CIFAR-100, Fast AutoAugment is slightly worse than AutoAugment and PBA except with Wide-ResNet-40-2, despite improving over baseline and Cutout.The authors attribute the gaps to insufficient policy exploration or over-training of model parameters.
- SVHN Results: On SVHN, Fast AutoAugment achieves comparable performance to AutoAugment and outperforms baseline and Cutout with Wide-ResNet-28-10.The experiment uses 1,000 randomly selected examples and the same settings as CIFAR.
5 Discussion
The discussion examines how policy-set size, transfer across model capacities, class-specific search, and Bayesian-optimization settings affect Fast AutoAugment. More sub-policies help up to a point, while transfer and search choices impose practical limitations.
- Sub-policy Count: Validation performance improves as the number of training sub-policies increases, up to approximately 100–125 sub-policies.This trend is tested with Wide-ResNet-40-2 and Wide-ResNet-28-10 on CIFAR-10 and CIFAR-100.
- Policy Transfer: Transferred policies have small performance gaps versus direct-search policies, and the gaps increase with model capacity.Policies searched with the smaller Wide-ResNet-40-2 remain better than default augmentations.
- Class-specific Search: Class-specific search on CIFAR-100 yields a 17.2% error rate for Wide-ResNet-28-10 using the 70 best-performing policies per class.The authors report no definite improvement over AutoAugment and Fast AutoAugment.
- Search Configuration: Tuning Bayesian-optimization meta-parameters such as search depth or kernel type does not significantly improve model performance empirically.These settings are considered during the augmentation search phase.
6 Conclusion
Fast AutoAugment automatically learns augmentation policies for a given task and convolutional network, with search significantly faster than AutoAugment. The paper identifies broader architectures, operations, and vision tasks as future directions.
- Fast AutoAugment automatically learns augmentation policies for a given task and convolutional neural network.
- Its search method is significantly faster than AutoAugment and outperforms human-crafted augmentation methods.
- Applying Fast AutoAugment to advanced architectures such as AmoebaNet would not increase search costs.
- Joint optimization of neural architecture search and Fast AutoAugment remains future work.
- Extending Fast AutoAugment beyond image classification to other computer vision tasks is planned for future work.