Source-linked AI summary
Towards Dropout Training for Convolutional Neural Networks
Haibing Wu, Xiaodong Gu
TL;DR
The paper addresses limited understanding of dropout in CNN convolutional and pooling layers. It interprets max-pooling dropout as multinomial sampling, introduces probabilistic weighted pooling for test-time averaging, and evaluates layer-specific strategies. Probabilistic weighted pooling outperforms max-pooling, while combining max-pooling and fully-connected dropout produces strong results across the reported datasets.
Problem
Dropout is established in fully-connected CNN layers, but its effects in convolutional and pooling layers remain insufficiently studied.
Method
The paper models max-pooling dropout as multinomial activation sampling and uses probabilistic weighted pooling to average possible dropout networks at test time.
Results
Probabilistic weighted pooling outperforms max-pooling and scaled max-pooling, while max-pooling and fully-connected dropout together achieve the strongest reported dataset results.
Takeaways & Limitations
Dropout design should consider pooling and fully-connected layers together, while convolutional dropout is beneficial but often less advantageous.
Takeaways & Limitations
The best dropout strategy remains open because network architecture, retaining probability, and training data strongly influence performance.
Abstract
from arXiv · showhide
Recently, dropout has seen increasing use in deep learning. For deep convolutional neural networks, dropout is known to work well in fully-connected layers. However, its effect in convolutional and pooling layers is still not clear. This paper demonstrates that max-pooling dropout is equivalent to randomly picking activation based on a multinomial distribution at training time. In light of this insight, we advocate employing our proposed probabilistic weighted pooling, instead of commonly used max-pooling, to act as model averaging at test time. Empirical evidence validates the superiority of probabilistic weighted pooling. We also empirically show that the effect of convolutional dropout is not trivial, despite the dramatically reduced possibility of over-fitting due to the convolutional architecture. Elaborately designing dropout training simultaneously in max-pooling and fully-connected layers, we achieve state-of-the-art performance on MNIST, and very competitive results on CIFAR-10 and CIFAR-100, relative to other approaches without data augmentation. Finally, we compare max-pooling dropout and stochastic pooling, both of which introduce stochasticity based on multinomial distributions at pooling stage.
1 Introduction
The paper examines dropout beyond fully-connected CNN layers, showing that max-pooling dropout has a multinomial interpretation and motivates probabilistic weighted pooling. It also reports competitive benchmark results and compares max-pooling dropout with stochastic pooling.
- Dropout works well in fully-connected CNN layers, but its effects in convolutional and pooling layers were not well studied.
- Max-pooling dropout is equivalent to sampling activations from a multinomial distribution with tunable retaining probability p.
- Probabilistic weighted pooling averages possible max-pooling-dropout networks at test time and outperforms max-pooling empirically.
- The number of possible max-pooling-dropout models grows exponentially with the number of hidden units entering pooling layers and decreases as pooling regions grow.
- Combining dropout in max-pooling and fully-connected layers achieves state-of-the-art MNIST results and competitive CIFAR-10 and CIFAR-100 results without data augmentation.
- Typical retaining probabilities often make max-pooling dropout outperform stochastic pooling by a large margin.
2 Review of Dropout Training for Convolutional Neural Networks
The review places dropout among regularization and model-averaging methods for deep CNNs, emphasizing shared parameters, approximate test-time averaging, and related stochastic approaches.
- CNNs use local connectivity and shared filters to reduce parameters, but large CNNs can still overfit small training datasets.
- Regularization alternatives include l2 penalties, early stopping, Bayesian fitting, weight elimination, and data augmentation.
- Dropout trains exponentially many possible shared-parameter models, while at most m×e models are explicitly trained across m examples and e epochs.
- At test time, weight scaling provides an efficient approximation to averaging exponentially many dropout sub-models.
- On CIFAR-10, fully-connected dropout reduced test error from 15.60% to 14.32%, while adding convolutional dropout reduced it further to 12.61%.
- Stochastic pooling randomly selects activations according to a multinomial distribution and uses probability weighting at test time.
- DropConnect randomly zeros weights rather than activations and often outperformed dropout on visual recognition datasets.
- Maxout networks combine grouped maximum linear responses with dropout and achieved strong results across MNIST, CIFAR-10, CIFAR-100, and SVHN.
3 Max-Pooling Dropout and Convolutional Dropout
The paper formalizes max-pooling dropout as multinomial activation sampling, proposes probabilistic weighted pooling for test-time averaging, and analyzes convolutional dropout. It finds that dropout choices interact with architecture and layer type.
- Max-Pooling Dropout at Training Time: Max-pooling dropout applies a Bernoulli mask before pooling, so max-pooling becomes stochastic rather than always selecting the strongest activation.
- Max-Pooling Dropout at Training Time: For ordered nonnegative activations, activation i is selected when larger activations are dropped and it is retained, yielding multinomial sampling probabilities.
- Max-Pooling Dropout at Training Time: Max-pooling dropout creates exponentially many possible models, with growth base b(t)=(t+1)^(1/t) that decreases as pooling-region size t increases.
- Max-Pooling Dropout at Test Time: At test time, scaled max-pooling selects the strongest activation and multiplies it by retaining probability p.
- Max-Pooling Dropout at Test Time: Probabilistic weighted pooling linearly weights activations by their training-time selection probabilities, matching the expected pooled output.
- Max-Pooling Dropout at Test Time: Probabilistic weighted pooling acts as model averaging because each sampled pooling index corresponds to a different model.
- Convolutional Dropout: Convolutional dropout masks feature-map activations before convolution, producing stochastic convolved features across local regions.
- Convolutional Dropout: Convolutional dropout improves test generalization but is often inferior to max-pooling or fully-connected dropout because CNN architecture already limits overfitting.
4 Empirical Evaluations
Experiments evaluate probabilistic weighted pooling, dropout placement, retaining probabilities, and comparisons with stochastic pooling across MNIST, CIFAR-10, and CIFAR-100. Probabilistic weighted pooling generally improves model averaging and test performance, while dropout placement and retaining probability materially affect results.
- Pooling at test time: Probabilistic weighted pooling produces lower training and test errors than scaled max-pooling and generalizes better than max-pooling without dropout on MNIST.The comparison covers two CNN architectures over 300 training epochs.
- Dropout in different layers: Convolutional dropout reduces over-fitting and test error, but fully-connected dropout is strongest on the smaller architecture and convolutional dropout is less advantageous overall.Max-pooling dropout performs well on both MNIST architectures, whereas combining convolutional and max-pooling dropout can increase error.
- Dropout in different layers: On MNIST, max-pooling and fully-connected dropout together achieve a 0.39% test error, improving on 0.81% without dropout and prior results of 0.45%-0.47%.Convolutional dropout reaches 0.60%, while max-pooling dropout alone reaches 0.47%.
- CIFAR experiments: On CIFAR-10 and CIFAR-100, probabilistic weighted pooling shows clear superiority over max-pooling and scaled max-pooling, while dropout in different layers improves generalization when appropriately combined.Improperly combining convolutional and max-pooling dropout can decrease performance.
- Comparison with stochastic pooling: Max-pooling dropout with typical retaining probabilities around 0.5 often outperforms stochastic pooling by a large margin, although performance follows a U-shaped relation with retaining probability.With probabilities that are too small or too large, max-pooling dropout can perform worse than stochastic pooling.
- Pooling at test time: Probabilistic weighted pooling generally outperforms max-pooling and scaled max-pooling for max-pooling-dropout-trained models, especially at small retaining probabilities.Increasing the retaining probability narrows the performance gap between pooling methods.
5 Conclusions
The paper studies dropout at max-pooling inputs, proposes probabilistic weighted pooling for test-time model averaging, and reports benefits across several datasets without data augmentation.
- Max-pooling dropout is equivalent to multinomial activation sampling during training, with exponentially many possible trained networks as pooling inputs increase.
- Probabilistic weighted pooling acts as test-time model averaging and outperforms max-pooling and scaled max-pooling empirically.
- Using max-pooling and fully-connected dropout together yields better MNIST results and comparable CIFAR-10 and CIFAR-100 performance against state-of-the-art methods without data augmentation.
- The comparison with stochastic pooling finds that its performance falls between results from max-pooling dropout configurations.