Source-linked AI summary
Max-Pooling Dropout for Regularization of Convolutional Neural Networks
Haibing Wu, Xiaodong Gu
TL;DR
Dropout works well in fully connected CNN layers, but its effect in pooling layers is less established. The paper characterizes max-pooling dropout as multinomial activation sampling and proposes probabilistic weighted pooling for test-time model averaging, with experiments validating its superiority over max-pooling.
Problem
The paper addresses limited understanding of dropout applied to inputs of max-pooling layers in convolutional neural networks.
Method
Max-pooling dropout is interpreted as multinomial activation sampling during training, and probabilistic weighted pooling uses the resulting probabilities for test-time model averaging.
Results
Probabilistic weighted pooling outperforms max-pooling and scaled max-pooling across tested retaining probabilities, while typical max-pooling-dropout probabilities often outperform stochastic pooling.
Takeaways & Limitations
The paper supports using probabilistic weighted pooling rather than max-pooling as a more accurate test-time approximation of averaging dropout-trained networks.
Abstract
from arXiv · showhide
Recently, dropout has seen increasing use in deep learning. For deep convolutional neural networks, dropout is known to work well in fully-connected layers. However, its effect in pooling layers is still not clear. This paper demonstrates that max-pooling dropout is equivalent to randomly picking activation based on a multinomial distribution at training time. In light of this insight, we advocate employing our proposed probabilistic weighted pooling, instead of commonly used max-pooling, to act as model averaging at test time. Empirical evidence validates the superiority of probabilistic weighted pooling. We also compare max-pooling dropout and stochastic pooling, both of which introduce stochasticity based on multinomial distributions at pooling stage.
1 Introduction
CNNs reduce overfitting through architectural properties and pooling, but large models can still overfit, motivating regularization beyond fully connected layers. This paper studies dropout in pooling layers and proposes probabilistic weighted pooling for test-time model averaging.
- Motivation: Large CNNs can still overfit relatively small training datasets despite fewer parameters and pooling-based translation invariance.CNNs benefit from local connectivity, shared filters, and pooling, but these properties do not eliminate overfitting.
- Motivation: Dropout combats overfitting by stochastically zeroing hidden activations and can be interpreted as efficient model averaging.The dropped units cannot influence retained units, breaking co-adaptations of feature detectors.
- Research gap: Dropout is established in fully connected CNN layers, whereas its effect in pooling layers remains insufficiently studied.The paper focuses on dropout applied to inputs of max-pooling layers.
- Contribution: Max-pooling dropout is equivalent to multinomial activation sampling, and probabilistic weighted pooling is proposed to average the resulting networks at test time.The retaining probability p tunes the sampling distribution.
- Empirical comparison: Typical retaining probabilities often make max-pooling dropout outperform stochastic pooling by large margins.The comparison is motivated by their shared multinomial sampling structure at the pooling stage.
2 Related Work
Prior work established dropout in fully connected layers and introduced related stochastic regularization methods, including stochastic pooling and maxout networks. These methods provide context for extending stochastic model averaging to pooling layers.
- Dropout in CNNs: Early CNN work used dropout in fully connected layers alongside data augmentation to reduce overfitting.The cited ImageNet work trained a large CNN on 1.2 million images and applied dropout only to fully connected layers.
- Stochastic pooling: Stochastic pooling randomly selects activations from each pooling region according to a multinomial distribution instead of always choosing the strongest activation.It is presented as a dropout-inspired regularization method.
- Other methods: Maxout networks combine with dropout to achieve strong results, while related stochastic model-averaging methods include drop-connect and adaptive dropout.The cited maxout comparison did not train maxout networks without dropout or directly compare rectified counterparts with dropout.
3 Max-Pooling Dropout
Max-pooling dropout applies a binary dropout mask before pooling, turning max-pooling into multinomial activation sampling during training. The paper then uses probabilistic weighted pooling at test time to approximate averaging across the possible dropout-trained models.
- 3.1 Max-Pooling Dropout at Training Time: Max-pooling dropout is equivalent to sampling an activation index from a multinomial distribution, including a zero-output event when all units are dropped.The selected activation becomes the pooled output.
- Pooling setup: Pooling combines activations within local regions, with max-pooling retaining only the strongest activation and average-pooling weighting all activations equally.The paper motivates max-pooling dropout as a way to introduce stochasticity while avoiding these contrasting behaviors.
- 3.1 Max-Pooling Dropout at Training Time: A Bernoulli mask independently drops pooling inputs before the modified activations are passed to the max-pooling operation.The masked activations are produced by element-wise multiplication with the original activations.
- 3.1 Max-Pooling Dropout at Training Time: With dropout, max-pooling need not select the strongest original activation, so training-time pooling becomes stochastic.The retained activation depends on which units survive in the pooling region.
- 3.1 Max-Pooling Dropout at Training Time: The number of possible max-pooling-dropout models is exponential in the number of pooling inputs, while its growth base decreases as pooling-region size increases.The base satisfies b(t)=(t+1)^(1/t) and is bounded above by 2.
- 3.2 Probabilistic Weighted Pooling at Test Time: Probabilistic weighted pooling linearly combines region activations using their multinomial selection probabilities to approximate averaging over all dropout models.It is proposed as a more accurate alternative to scaled max-pooling at test time.
4 Empirical Evaluations
Experiments on MNIST, CIFAR-10, and CIFAR-100 compare probabilistic weighted pooling with max-pooling, scaled max-pooling, and stochastic pooling. Probabilistic weighted pooling generally performs best, while max-pooling dropout with typical retaining probabilities outperforms stochastic pooling.
- Experimental Setup: Experiments use MNIST, CIFAR-10, and CIFAR-100, with CNNs trained using different max-pooling-dropout retaining probabilities.The models use ReLU activations and stochastic mini-batch gradient descent; dataset-specific architectures are evaluated.
- Probabilistic Weighted Pooling vs. (Scaled) Max-Pooling: On MNIST, probabilistic weighted pooling generally outperforms max-pooling and scaled max-pooling at test time.The advantage is especially substantial for small retaining probabilities, while the gap narrows as p increases.
- Probabilistic Weighted Pooling vs. (Scaled) Max-Pooling: On CIFAR-10 and CIFAR-100, probabilistic weighted pooling is the best performer across retaining probabilities p = 0.3, 0.5, and 0.7.The performance gap among pooling methods narrows as the retaining probability increases.
- Max-Pooling Dropout vs. Stochastic Pooling: Max-pooling dropout samples activations from a multinomial distribution during training, while probabilistic weighted pooling averages activations using probabilistic weights at test time.The test-time weighting is used to approximate model averaging over networks trained with max-pooling dropout.
- Max-Pooling Dropout vs. Stochastic Pooling: Max-pooling dropout performance follows a U-shape with retaining probability, and typical p around 0.5 outperforms stochastic pooling by a large margin.When p is too small or too large, max-pooling dropout performs worse than stochastic pooling.
5 Conclusions
The paper analyzes dropout applied before max-pooling in convolutional neural networks and interprets it as multinomial activation sampling during training. It proposes probabilistic weighted pooling for test-time model averaging, finds it superior to max-pooling and scaled max-pooling, and reports that stochastic pooling performs between max-pooling-dropout models with different retaining probabilities.
- Conclusions: The paper addresses understanding and using dropout on inputs to max-pooling layers in convolutional neural networks.It characterizes the resulting stochastic pooling behavior and its model-averaging interpretation.
- Conclusions: Max-pooling dropout is equivalent to randomly selecting activations according to a multinomial distribution during training.The number of possibly trained networks is exponential in the number of pooling-layer input units.
- Conclusions: Probabilistic weighted pooling is proposed for test-time model averaging and is superior to max-pooling and scaled max-pooling in the experiments.The method uses the stochastic training interpretation to average possible max-pooling-dropout-trained networks.
- Conclusions: Stochastic pooling performs between max-pooling-dropout models trained with different retaining probabilities.The comparison is empirical and concerns pooling methods with stochasticity at the pooling stage.