Source-linked AI summary
Fractional Max-Pooling
Benjamin Graham
TL;DR
Standard max-pooling rapidly shrinks hidden layers and uses disjoint regions, motivating a gentler pooling strategy. The paper introduces stochastic fractional max-pooling, which uses non-integer reduction factors and varied pooling regions. Across several datasets, the authors report substantial performance improvements, including better results for CIFAR-100 without dropout.
Problem
Standard max-pooling rapidly reduces hidden-layer size and uses disjoint regions, limiting deeper convolutional-network designs and generalization.
Method
Fractional max-pooling reduces spatial size by a factor α with 1 < α < 2 and randomizes the choice of pooling regions.
Results
Fractional max-pooling produces substantial performance improvements across several datasets, including improved CIFAR-100 performance without dropout.
Takeaways & Limitations
Overlapping FMP performs better than disjoint FMP, while pseudorandom regions perform better than random regions when training-data augmentation is used.
Takeaways & Limitations
The distortions created by random pooling are decomposable into separate x-axis and y-axis distortions, leaving more general distortion patterns unexplored.
Abstract
from arXiv · showhide
Convolutional networks almost always incorporate some form of spatial pooling, and very often it is alpha times alpha max-pooling with alpha=2. Max-pooling act on the hidden layers of the network, reducing their size by an integer multiplicative factor alpha. The amazing by-product of discarding 75% of your data is that you build into the network a degree of invariance with respect to translations and elastic distortions. However, if you simply alternate convolutional layers with max-pooling layers, performance is limited due to the rapid reduction in spatial size, and the disjoint nature of the pooling regions. We have formulated a fractional version of max-pooling where alpha is allowed to take non-integer values. Our version of max-pooling is stochastic as there are lots of different ways of constructing suitable pooling regions. We find that our form of fractional max-pooling reduces overfitting on a variety of datasets: for instance, we improve on the state-of-the art for CIFAR-100 without even using dropout.
1 Convolutional neural networks
Convolutional networks combine convolutional filters with spatial pooling, but standard max-pooling can shrink representations too quickly and use disjoint regions. Fractional max-pooling addresses these limitations by reducing spatial size by a non-integer factor and randomizing pooling-region selection.
- Convolutional networks combine layers of convolutional filters with spatial pooling such as max-pooling.
- 2 × 2 max-pooling is popular because it is fast, rapidly reduces hidden-layer size, and encodes invariance to translations and elastic distortions.
- Standard max-pooling can limit generalization through disjoint regions and rapidly reduce hidden-layer size, requiring back-to-back convolutional layers for deeper networks.
- Because existing alternatives still halve hidden-layer dimensions, the paper asks whether pooling can reduce spatial size more gently.
- Fractional max-pooling reduces spatial size by α with 1 < α < 2 and randomizes pooling-region choice rather than the operation within each region.
- FMP implementation varies the pooling fraction, whether regions are random or pseudorandom, and whether they are disjoint or overlapping.
2 Fractional max-pooling
Fractional max-pooling constructs pooling regions from integer sequences whose increments are one or two, supporting disjoint or overlapping arrangements with fractional spatial reduction. Pseudorandom sequences produce more stable regions, while random regions create elastic distortions in repeatedly pooled images.
- Max-pooling partitions an input matrix into pooling regions and outputs the maximum value from each region.
- FMP can reduce spatial-size decay n times more slowly than conventional pooling, while ratios above two remain available when speed matters more.
- Pooling arrangements include overlapping squares and disjoint collections of rectangles, with multiple divisions possible for a fixed input and output size.
- The pooling regions are generated from increasing integer sequences with increments of one or two, using either random or pseudorandom sequences.
- Pseudorandom regions are more stable than random ones; repeated disjoint random pooling produces elastic distortion, whereas pseudorandom pooling faithfully scales images.
3 Implementation
The networks are trained using an implementation of a sparse convolutional network.
- The networks are trained using an implementation of a sparse convolutional network.
- The implementation allows convolutional-network training within the paper’s experimental setup.
- The passage identifies the training implementation as sparse rather than describing its further mechanics.
C2 C2 FMP
FMP networks use changing pooling-region configurations during training and testing, allowing model averaging across related networks. Repeated test-time classification with majority voting can improve accuracy.
- Model averaging: Each training or testing pass uses a different random or pseudorandom pooling-region sequence, creating an ensemble of related networks.The authors compare this ensemble effect to dropout, where different masks define related networks.
- Model averaging: Model averaging can improve FMP performance, analogous to averaging models induced by different dropout masks.
- Model averaging: Majority voting over repeated classifications of the same test image can substantially improve accuracy.Figure 4 reports the effect of repeat testing for one MNIST-trained FMP network.
4 Results
Across MNIST, CIFAR-100, Assamese, Chinese handwriting, and CIFAR-10, FMP achieves strong test performance, with repeated testing often improving reported error. Results also show effects of network size, pooling design, and augmentation.
- Without training set augmentation or dropout: 27.62% / 23.82% test errors result from 96nC2 with dropout and slower learning-rate decay, compared with 35.09% / 29.66% for 32nC2.
- Assamese handwriting: FMP without data augmentation outperforms MP2 with training data augmentation on Assamese handwriting, linking pooling and augmentation as alternative ways to encode distortion invariance.
- Without training set augmentation or dropout: 4.50%, 3.67%, and 3.47% test errors are obtained on CIFAR-10 with one, twelve, and one hundred tests using pseudorandom overlapping FMP.
5 Conclusions
The authors report substantial performance improvements from fractional max-pooling across several datasets. They also identify pooling-design and regularization trade-offs, and suggest exploring distortions beyond the current construction.
- Conclusions: Fractional max-pooling produces substantial performance improvements across several popular datasets.
- Conclusions: Overlapping FMP appears better than disjoint FMP, while pseudorandom regions appear better than random regions when training-data augmentation is used.
- Conclusions: The distortion created by random pooling is decomposable into separate x-axis and y-axis distortions.The authors propose exploring pooling regions that cannot be represented by the current equation to encode more general distortions.