Source-linked AI summary
DropBlock: A regularization method for convolutional networks
Golnaz Ghiasi, Tsung-Yi Lin, Quoc V. Le
TL;DR
Dropout is less effective for convolutional layers because spatially correlated features can preserve information, motivating structured regularization. The paper introduces DropBlock, which drops contiguous feature-map regions, and reports stronger regularization than dropout across ImageNet and COCO experiments.
Problem
Dropout is less effective in convolutional layers because spatially correlated features can still transmit input information, allowing networks to overfit.
Method
DropBlock regularizes CNNs by dropping spatially contiguous regions of feature maps rather than independent random units.
Results
DropBlock outperforms dropout across models and datasets, including 78.13% ImageNet accuracy for ResNet-50 and COCO RetinaNet AP improving from 36.8% to 38.4%.
Takeaways & Limitations
Applying DropBlock in skip connections and gradually increasing dropped units during training improves accuracy and robustness to hyperparameter choices.
Abstract
from arXiv · showhide
Deep neural networks often work well when they are over-parameterized and trained with a massive amount of noise and regularization, such as weight decay and dropout. Although dropout is widely used as a regularization technique for fully connected layers, it is often less effective for convolutional layers. This lack of success of dropout for convolutional layers is perhaps due to the fact that activation units in convolutional layers are spatially correlated so information can still flow through convolutional networks despite dropout. Thus a structured form of dropout is needed to regularize convolutional networks. In this paper, we introduce DropBlock, a form of structured dropout, where units in a contiguous region of a feature map are dropped together. We found that applying DropbBlock in skip connections in addition to the convolution layers increases the accuracy. Also, gradually increasing number of dropped units during training leads to better accuracy and more robust to hyperparameter choices. Extensive experiments show that DropBlock works better than dropout in regularizing convolutional networks. On ImageNet classification, ResNet-50 architecture with DropBlock achieves $78.13\%$ accuracy, which is more than $1.6\%$ improvement on the baseline. On COCO detection, DropBlock improves Average Precision of RetinaNet from $36.8\%$ to $38.4\%$.
1 Introduction
Dropout is less effective in convolutional layers because spatially correlated features can preserve input information, motivating structured regularization. DropBlock drops contiguous feature-map regions and outperforms dropout across reported classification and detection experiments.
- Motivation: Spatially correlated convolutional features let information reach later layers despite random dropout, contributing to overfitting.This motivates a more structured form of dropout for convolutional networks.
- Method: DropBlock drops features in contiguous feature-map blocks so networks must look elsewhere for evidence.The method is introduced as a structured form of dropout for convolutional networks.
- Results: 78.13% ImageNet accuracy was achieved by ResNet-50 with DropBlock, improving the baseline from 76.51%.The reported improvement is 1.62 percentage points.
- Results: 38.4% COCO AP was achieved by RetinaNet with DropBlock, improving the baseline from 36.8%.The reported improvement is 1.6 percentage points.
2 Related work
Dropout-inspired regularizers inject noise to reduce overfitting, while convolutional networks generally benefit from architecture-specific structured noise. DropBlock is presented alongside methods that structure noise at different network levels, and its schedule is related to ScheduledDropPath.
- Noise injection: Dropout-inspired methods inject noise into neural networks to reduce overfitting.The related methods include DropConnect, maxout, StochasticDepth, DropPath, ScheduledDropPath, shake-shake, and ShakeDrop regularization.
- Convolutional networks: Successful regularization methods for convolutional neural networks generally require structured noise.DropPath exemplifies this by zeroing an entire layer.
- Related architectures: Recurrent networks likewise use specialized noise-injection methods, including Variational Dropout and ZoneOut.These methods target recurrent connections.
- DropBlock relation: DropBlock generalizes Cutout by applying input-style block zeroing at every feature map in a convolutional network.The experiments report that a linearly increasing DropBlock ratio is more robust than a fixed ratio during training.
3 DropBlock
DropBlock samples spatial masks and expands sampled zero positions into contiguous blocks, then masks and normalizes layer activations. Its behavior is controlled mainly by block_size and γ, with scheduled dropping improving robustness during training.
- DropBlock: DropBlock drops contiguous regions of a feature map rather than independent random units.Its two main parameters are block_size, the dropped block size, and γ, which controls how many activation units are dropped.
- Masking: Each feature channel can receive its own DropBlock mask, which worked better experimentally than sharing one mask across channels.Algorithm 1 corresponds to the per-channel masking variant.
- Mask sampling: A sampled zero in M expands into a block_size × block_size zero block before the mask is applied to activations.The algorithm then normalizes features using count(M)/count_ones(M).
- Inference: DropBlock is not applied during inference, which is interpreted as evaluating an averaged prediction across exponentially many subnetworks.The covered subnetworks include models that do not see contiguous feature-map regions.
- Hyperparameters: block_size = 1 makes DropBlock resemble dropout, while a block covering the full feature map makes it resemble SpatialDropout.The implementation uses a constant block_size across feature-map resolutions.
- Hyperparameters: γ is adjusted for block expansion and valid seed-region size rather than set directly.The approximation accounts for overlapping blocks; keep_prob is first estimated between 0.75 and 0.95.
- Scheduled DropBlock: Gradually decreasing keep_prob from 1 to a target value is more robust than using a fixed keep_prob during training.The experiments use a linear schedule that works across many hyperparameter settings.
4 Experiments
Experiments evaluate DropBlock across image classification, detection, and segmentation, comparing placements, block sizes, schedules, and alternative regularizers. Results show gains over dropout and transfer across architectures and tasks, with larger blocks and scheduled keep_prob performing especially well.
- Experimental scope: Experiments cover ImageNet classification with ResNet-50 and AmoebaNet, COCO detection with RetinaNet, and semantic segmentation from pretrained or random initialization.The study also evaluates DropBlock placement and configuration choices within these settings.
- ResNet-50 classification: 1.3% top-1 accuracy separates DropBlock from dropout in the ResNet-50 ImageNet comparison.The comparison applies DropBlock to groups 3 and 4, with block_size = 7 by default; scheduled keep_prob further affects performance.
- ResNet-50 classification: block_size = 7 applied to groups 3 and 4 gives the best DropBlock configuration in the ResNet-50 sweep.Larger blocks generally outperform block_size = 1, and DropBlock shows a large gain over the best dropout result.
- Comparisons: SpatialDropout improves over dropout but remains inferior to DropBlock, while Cutout does not improve ImageNet accuracy in these experiments.SpatialDropout can be too harsh on the higher-resolution group 3 feature map.
- AmoebaNet: 82.52% top-1 accuracy is reported for AmoebaNet-B with DropBlock, compared with 82.25% without it.DropBlock is applied after batch normalization and in skip connections of the last 50% of cells.
- Experimental analysis: DropBlock-trained models are more robust to inference-time block-size changes, and CAMs show attention across several spatially distributed regions.The block_size = 7 model retains the benefit of block_size = 1 more effectively than the reverse configuration.
5 Discussion
DropBlock is presented as an effective CNN regularizer that consistently outperforms dropout, with stronger robustness and more spatially distributed representations. Increasing dropped units during training and applying DropBlock in skip connections further improves accuracy.
- 5 Discussion: DropBlock consistently outperforms dropout as a regularizer across extensive experiments.The paper evaluates it in ImageNet classification and COCO detection.
- 5 Discussion: Class activation mapping suggests DropBlock regularizes more spatially distributed representations.
- 5 Discussion: Applying DropBlock in skip connections in addition to convolution layers increases accuracy.
- 5 Discussion: Gradually increasing the number of dropped units during training improves accuracy and robustness to hyperparameter choices.