Source-linked AI summary
Improved Regularization of Convolutional Neural Networks with Cutout
Terrance DeVries, Graham W. Taylor
TL;DR
Convolutional neural networks can overfit because their representational power makes generalization difficult. The paper introduces cutout, which randomly masks square input regions during training, and reports state-of-the-art results on CIFAR-10, CIFAR-100, and SVHN, including 2.56%, 15.20%, and 1.30% test error.
Problem
Convolutional neural networks’ increased representational power also increases their probability of overfitting, leading to poor generalization.
Method
Cutout regularizes convolutional neural networks by randomly removing contiguous square regions from input images during training.
Results
2.56%, 15.20%, and 1.30% test error were achieved on CIFAR-10, CIFAR-100, and SVHN, respectively, yielding state-of-the-art benchmark performance.
Takeaways & Limitations
Cutout is complementary to existing data augmentation and regularization techniques and improves performance on modern convolutional architectures.
Abstract
from arXiv · showhide
Convolutional neural networks are capable of learning powerful representational spaces, which are necessary for tackling complex learning tasks. However, due to the model capacity required to capture such representations, they are often susceptible to overfitting and therefore require proper regularization in order to generalize well. In this paper, we show that the simple regularization technique of randomly masking out square regions of input during training, which we call cutout, can be used to improve the robustness and overall performance of convolutional neural networks. Not only is this method extremely easy to implement, but we also demonstrate that it can be used in conjunction with existing forms of data augmentation and other regularizers to further improve model performance. We evaluate this method by applying it to current state-of-the-art architectures on the CIFAR-10, CIFAR-100, and SVHN datasets, yielding new state-of-the-art results of 2.56%, 15.20%, and 1.30% test error respectively. Code is available at https://github.com/uoguelph-mlrg/Cutout
1. Introduction
CNNs learn complex image representations but their increased representational power raises overfitting and poor-generalization risks. Cutout addresses this by masking contiguous input regions, encouraging reliance on broader image context and working alongside established architectures and regularizers.
- CNNs’ increased representational power also increases the probability of overfitting, leading to poor generalization.
- Cutout drops contiguous input sections only at the CNN input layer, rather than dropping individual pixels or intermediate feature units.
- Removed regions propagate through subsequent feature maps, leaving no trace except information recoverable from surrounding context.This encourages networks to use the full image context instead of relying on a small set of specific visual features.
- Cutout can improve CNN robustness and performance while working with residual networks, batch normalization, dropout, and data augmentation.
2. Related Work
The paper situates cutout among data augmentation, dropout, and image-corruption methods. It distinguishes cutout by directly applying contiguous-region masking during supervised training.
- Data augmentation: Data augmentation improves CNN accuracy and robustness through transformations such as affine changes, mirroring, cropping, and altered color or intensity.Prior work also uses elastic deformation, blur, noise, occlusions, geometric distortions, stretching, and learned sample combinations.
- Dropout: Dropout regularizes networks by randomly setting hidden activations to zero during training and scaling outputs at evaluation.It approximately averages over an exponential number of smaller subnetworks and acts as a robust form of bagging.
- Dropout: Dropout is less powerful in convolutional layers because they have fewer parameters and neighboring pixels share information.Proposed alternatives include SpatialDropout, probabilistic weighted pooling, and max-drop, with performance depending on architecture and batch normalization.
- Image corruption methods: Denoising auto-encoders and context encoders corrupt images for self-supervised reconstruction, erasing individual pixels or larger spatial regions.These models use remaining pixels as context to reconstruct missing content.
- Image corruption methods: Contiguous input removal had been used for image corruption in context encoders but, to the authors’ knowledge, not directly for supervised model training.Cutout instead uses zero-masking to completely obstruct an entire region, unlike scratches, dots, or scribbles overlaying characters.
3. Cutout
Cutout regularizes CNNs by randomly masking contiguous square regions of input images, creating partially occluded training examples. Unlike intermediate-layer dropout variants, it removes visual content at the input stage so subsequent feature maps consistently lack the masked regions.
- Method: Cutout removes contiguous sections of input images, augmenting the dataset with partially occluded versions of existing samples.It can be interpreted as dropout extended to input space with a spatial prior.
- Method: Unlike dropout variants that drop units in intermediate layers, cutout masks the input so removed visual features disappear from all subsequent feature maps.Other variants generally treat feature maps individually, allowing removed features to remain in other maps.
- Motivation: By simulating object occlusion, cutout prepares models for real-world occlusions and encourages decisions based on more complete image context.The motivation applies to tasks including object recognition, tracking, and human pose estimation.
- Method: Randomly removing fixed-size regions performed as well as targeted removal while avoiding feature-map manipulation, so experiments used fixed-size regions.The targeted approach aimed to remove maximally activated features, whereas the simpler alternative selected regions randomly.
- Implementation: During each training epoch, cutout applies a fixed-size zero-mask at a random image location without test-time weight rescaling.The authors use square patches because region size matters more than shape; zero-centered normalization is recommended.
4. Experiments
Cutout was evaluated across CIFAR-10, CIFAR-100, SVHN, and STL-10 using modern residual architectures and validation-based patch-size selection. It improved test performance across datasets, including a new 1.30% SVHN test error and gains on STL-10 with and without augmentation.
- Experimental setup: Cutout was evaluated on CIFAR-10, CIFAR-100, SVHN, and STL-10 using ResNet and WideResNet architectures.CIFAR experiments used ResNet18 and WRN-28-10; SVHN and STL-10 used WideResNet variants.
- Cutout-size selection: Validation-based grid searches selected square cutout sizes, including 8 × 8 pixels for CIFAR-100, 20 × 20 pixels for SVHN, and 24 × 24 or 32 × 32 pixels for STL-10.The optimal cutout size decreased as the number of classes increased, while STL-10 used different sizes with and without data augmentation.
- CIFAR results: 0.4 to 2.0 percentage points: cutout improved CIFAR-10 and CIFAR-100 accuracy for ResNet18 and WRN-28-10.These gains occurred despite batch normalization, dropout, and standard data augmentation already being used.
- SVHN results: 1.30% test error: cutout reduced WRN-16-8 test error on SVHN by an average of 0.3 percentage points.The result improved on the configuration’s prior 1.54% test error and was reported as new state-of-the-art performance.
- STL-10 results: 2.7 percentage points and 1.5 percentage points: cutout reduced STL-10 test error without and with data augmentation, respectively.The selected square sizes were 24 × 24 pixels without augmentation and 32 × 32 pixels with augmentation.
- Activation analysis: Cutout increased shallow-layer activation strength and produced more deep-layer activations in the distribution tail, indicating broader feature use for predictions.The activation analysis was averaged over all test-set samples after sorting activations by ascending magnitude within each layer.
5. Conclusion
Cutout was conceived to remove highly activated visual features and encourage reliance on complementary, less prominent features for robustness to occlusion. Experiments found that randomly masking square image regions, a simpler approach, performed equivalently to the targeted method.
- Method motivation: Cutout was originally conceived as a targeted method for removing visual features with high activations in later CNN layers.Its motivation was to encourage networks to focus on complementary and less prominent features.
- Method motivation: The intended benefit was improved generalization to situations such as occlusion through reliance on less prominent visual features.
- Experimental conclusion: Randomly masking square image regions performed equivalently in the conducted experiments while being conceptually and computationally simpler.
- Feature activations: Figure 4 compares feature-activation magnitudes for standard ResNet18 and cutout-trained ResNet18 models at three depths.The activations are sorted by descending value and averaged over all test samples.
A. Supplementary Materials
Figure 5 compares feature-activation magnitudes between a standard ResNet18 and a cutout-trained ResNet18 across three depths and multiple test samples.
- Feature activations are sorted by descending magnitude in Figure 5.
- Each row represents a different test sample.
- The comparison includes a standard ResNet18 and a ResNet18 trained with cutout at three depths.