Source-linked AI summary
Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
Matthew D. Zeiler, Rob Fergus
TL;DR
Large convolutional networks can overfit, while dropout has limited benefits in convolutional layers. The paper replaces deterministic pooling with activation-weighted stochastic sampling, and reports state-of-the-art performance across multiple image benchmarks without data augmentation.
Problem
Large convolutional networks remain prone to overfitting, and dropout does not seem to provide the same benefits for convolutional layers.
Method
The method replaces deterministic pooling with sampling from a multinomial distribution defined by activations within each pooling region, using probability weighting at test time.
Results
The method achieves state-of-the-art performance on numerous datasets relative to approaches without data augmentation, including gains of 0.47% on CIFAR-10, 2.66% on CIFAR-100, and 2.10% on SVHN.
Takeaways & Limitations
Stochastic pooling can be combined with weight decay, dropout, and data augmentation, with negligible computational overhead and no hyperparameters to tune.
Abstract
from arXiv · showhide
We introduce a simple and effective method for regularizing large convolutional neural networks. We replace the conventional deterministic pooling operations with a stochastic procedure, randomly picking the activation within each pooling region according to a multinomial distribution, given by the activities within the pooling region. The approach is hyper-parameter free and can be combined with other regularization approaches, such as dropout and data augmentation. We achieve state-of-the-art performance on four image datasets, relative to other approaches that do not utilize data augmentation.
1 Introduction
The paper addresses overfitting in high-capacity convolutional networks by introducing stochastic pooling as a regularizer that samples activations instead of deterministically pooling them.
- High-capacity neural networks are prone to overfitting, motivating regularization methods that enable larger models with better test performance.
- Dropout improves many tasks but provides weaker benefits in convolutional layers used for vision.
- The proposed method makes pooling stochastic by drawing an activation from a multinomial distribution formed by activations in each pooling region.
- Stochastic pooling can be viewed as max pooling over many locally deformed copies of an input image, with multilayer selection yielding exponentially many deformations.
2 Review of Convolutional Networks
Convolutional networks alternate convolution and pooling to build increasingly transformation-invariant features, but conventional average and max pooling have distinct drawbacks.
- Convolutional networks alternate convolution and pooling, producing pooled feature maps from local image regions.
- Pooling reduces sensitivity to the precise locations of image structures, supporting increasingly invariant features for classification.
- Rectified linear activations are especially suited to stochastic pooling because they produce nonnegative responses and zeros within pooling regions.
- Average pooling includes every element, allowing low-magnitude or zero responses to down-weight strong activations and permitting cancellation with tanh nonlinearities.
- Max pooling selects the largest element but can overfit training data, whereas the proposed stochastic scheme retains max-pooling advantages while helping prevent overfitting.
3 Stochastic Pooling
Stochastic pooling samples pooled activations according to normalized regional responses during training and uses probability-weighted averaging at test time, approximating model averaging without instantiating many models.
- For each pooling region, stochastic pooling normalizes activations into multinomial probabilities and samples a location from that distribution.
- Independent samples are drawn across pooling regions and layers, and the selected location routes the gradient during backpropagation as in max pooling.
- Unlike max pooling, stochastic pooling can utilize non-maximal activations within a region, representing multimodal activation distributions.
- At test time, probability-weighted averaging replaces stochastic sampling because prediction noise from sampling degrades performance.
- The weighting differs from standard averaging because elements receive activation-dependent weights and the denominator is the sum of activations rather than region size.
- Probability weighting approximates averaging over n^d possible models without instantiating them, where d is the number of regions and n their size.
- Using probabilities for training-time sampling and test-time weighting leads to state-of-the-art performance on common benchmarks.
4 Experiments
Experiments compare stochastic pooling with average and max pooling across image-classification settings. On CIFAR-10, stochastic pooling reduced overfitting and surpassed a no-data-augmentation state-of-the-art result.
- The experiments compare average, max, and stochastic pooling using mini-batch gradient descent with momentum across image-classification tasks.The baseline architecture uses three convolutional layers with 5x5 filters, 64 feature maps per layer, and rectified linear units.
- Stochastic pooling avoids overfitting unlike average and max pooling and achieves fewer test errors on CIFAR-10.Its training and test-error progression is evaluated over 280 epochs.
- 0.47% improvement over the CIFAR-10 state-of-the-art result was achieved without the additional locally connected layer.The comparison uses an approach with no data augmentation but dropout on an additional locally connected layer.
- 3x3 pooling regions appear optimal for stochastic pooling on CIFAR-10, while smaller regions overfit and larger regions may be noisier during training.Stochastic pooling outperforms max and average pooling at all tested sizes: 5x5, 4x4, 3x3, and 2x2.
4.3 MNIST
On MNIST, stochastic pooling avoided the complete overfitting observed with max pooling and outperformed methods that did not use data augmentation. The experiments compare pooling approaches against state-of-the-art convolutional methods.
- Stochastic pooling and max pooling errors dropped quickly during MNIST training, but max pooling completely overfit the training data.Weight decay prevented average pooling from overfitting, although its performance was inferior to the other two methods.
- Stochastic pooling outperforms other methods that do not use data augmentation such as jittering or elastic distortions.The cited state-of-the-art single-model approach uses elastic distortions to augment the original training set.
- The MNIST comparison covers various pooling methods, with the first two table rows representing current state-of-the-art approaches.
4.4 CIFAR-100
On CIFAR-100, stochastic pooling outperforms conventional pooling methods and surpasses the reported state-of-the-art method by 2.66%.
- Stochastic pooling outperforms conventional pooling methods on CIFAR-100 by preventing over-fitting.The dataset contains 100 classes and 50,000 training examples, with 500 examples per class.
- 2.66%: stochastic pooling surpasses the reported state-of-the-art method based on receptive field learning.
4.5 Street View House Numbers
On SVHN, stochastic pooling prevents overfitting in a large convolutional network and beats the existing state-of-the-art result by 2.10%, a relative gain of 43%.
- 604,388 training images and 26,032 test images define the SVHN evaluation setting.The task classifies the digit centered in each cropped 32x32 color image.
- Local contrast normalization on each RGB channel normalized brightness and color variations and helped training proceed quickly.Subtracting the per-pixel mean did not sufficiently modify the image statistics.
- Stochastic pooling prevented overfitting in a 64-64-128 model trained for 500 epochs despite substantial training data.
- 2.10%: stochastic pooling beat the existing state-of-the-art multi-stage convolutional network, a relative gain of 43%.
- Table 4 compares pooling methods in models with 64 or 128 layer-3 feature maps against state-of-the-art results with and without data augmentation.
4.6 Reduced Training Set Size
Across reduced training-set sizes on MNIST and CIFAR-10, stochastic pooling generally overfits less than other pooling approaches.
- Stochastic pooling generally overfits less than other pooling approaches when training on reduced MNIST and CIFAR-10 datasets.The evaluated training sizes range from 1,000 examples to the full training set.
- Figure 5 reports test error for training-set sizes of 1,000, 2,000, 3,000, 5,000, 10,000, half, and the full dataset.
4.7 Importance of Model Averaging
The experiments show that stochastic pooling is needed during training, while probability weighting is preferred at test time; averaging sampled models approaches probability weighting but costs N-fold more computation.
- Test-time stochastic sampling slightly degrades performance but still outperforms max or average pooling at test time.
- As N increases, sampled-model results approach probability weighting, but computation increases N-fold.
- Models trained with max or average pooling perform poorly when stochastic pooling is introduced only at test time.Training with stochastic pooling incorporates non-maximal elements and sampling noise, making the model more robust at test time.
- Probability weighting during training causes easy overfitting and suboptimal test performance, whereas its test-time benefits require compatible training.
- The best CIFAR-10 performance uses stochastic pooling during training and probability weighting at test time.This combination is described as preventing over-fitting while providing the best test-time aggregation.
4.8 Visualizations
The visualizations use a deconvolutional network to map feature maps back to input pixels and compare feed-forward with uniform pooling-location sampling. Feed-forward sampling preserves data regularities more effectively than replacing pooling distributions with uniform ones.
- 4.8 Visualizations: The deconvolutional network inverts pooling and filtering to decode top-layer feature maps back to input pixels.Its unpooling operation uses the locations selected during the forward pass.
- 4.8 Visualizations: The comparison samples pooling locations either from feed-forward distributions or from uniform distributions.Figures 6(b–h) replace one or more pooling layers’ feed-forward distributions with uniform distributions.
- 4.8 Visualizations: Feed-forward multinomial distributions effectively capture regularities in the data despite the model’s stochasticity.The paper uses deconvolutional outputs to demonstrate this comparison.
5 Discussion
The authors present stochastic pooling as a simple, compatible regularization strategy and report that it can be inserted into existing convolutional architectures. The figure illustrates multiple stochastic pooling instantiations and alternative sampling distributions.
- 5 Discussion: Stochastic pooling combines with weight decay, dropout, and data augmentation to prevent overfitting in deep convolutional networks.The strategy is presented as compatible with other regularization methods.
- 5 Discussion: The method achieves state-of-the-art performance on numerous datasets relative to approaches that do not use data augmentation.The comparison scope excludes approaches employing data augmentation.
- 5 Discussion: Stochastic pooling has negligible computational overhead and no hyper-parameters to tune, allowing it to replace pooling in existing architectures.The paper describes the method as swappable into any existing convolutional network architecture.
- 5 Discussion: Figure 6 shows repeated stochastic pooling instantiations for a horse image, alongside max and average pooling visualizations.Each image in a 4×4 block represents one pooling-location instantiation.