Source-linked AI summary
Pooling Methods in Deep Neural Networks, a Review
Hossein Gholamalinezhad, Hossein Khosravi
TL;DR
Pooling reduces feature-map dimensionality in convolutional systems, while pooling methods involve trade-offs such as information preservation and loss of finer details. This paper reviews pooling methods from 1989 to 2020, dividing them into popular and novel methods and describing each.
Problem
Pooling reduces the dimensionality of feature maps, but pooling choices can involve information loss and differing treatment of feature representations.
Method
The paper reviews famous and useful pooling methods from 1989 to 2020, dividing them into popular and novel methods and describing each.
Results
The review describes pooling methods spanning popular and novel categories.
Takeaways & Limitations
Pooling methods differ in how they preserve information and manage representation details during dimensionality reduction.
Takeaways & Limitations
Using one large pooling window can lead to the loss of finer details.
Abstract
from arXiv · showhide
Nowadays, Deep Neural Networks are among the main tools used in various sciences. Convolutional Neural Network is a special type of DNN consisting of several convolution layers, each followed by an activation function and a pooling layer. The pooling layer is an important layer that executes the down-sampling on the feature maps coming from the previous layer and produces new feature maps with a condensed resolution. This layer drastically reduces the spatial dimension of input. It serves two main purposes. The first is to reduce the number of parameters or weights, thus lessening the computational cost. The second is to control the overfitting of the network. An ideal pooling method is expected to extract only useful information and discard irrelevant details. There are a lot of methods for the implementation of pooling operation in Deep Neural Networks. In this paper, we reviewed some of the famous and useful pooling methods.
1. Introduction
The introduction presents pooling as a key CNN operation that downsamples feature maps, reduces computational demands, and helps control overfitting. It motivates a review of pooling methods organized into popular and novel categories.
- Deep-learning context: The introduction situates CNNs among DNN architectures commonly used for classification and object segmentation.CNNs are described as consisting of convolution, pooling, and fully connected layers, with flattening preparing feature maps for classification.
- Pooling in CNNs: Pooling reduces computational complexity by eliminating some connections between convolutional layers and upper layers.Reducing feature-map size and network parameters lowers the computational cost.
- Pooling in CNNs: Pooling reduces feature-map dimensionality by combining multiple values into fewer values.It performs down-sampling and produces feature maps with condensed resolution.
- Pooling in CNNs: Pooling keeps useful information while eliminating irrelevant information from feature representations.The introduction describes this selective compression as an expected property of an ideal pooling method.
- Pooling in CNNs: Pooling operators also provide a form of spatial transformation invariance.
- Review scope: The review divides pooling methods into popular and novel categories and discusses representative methods in each.Popular methods include Average, Max, Mixed, L_P, Stochastic, Spatial Pyramid, and Region of Interest Pooling; novel methods include newer approaches such as Spectral and Genetic-based Pooling.
2. Popular Pooling Methods
Popular pooling methods down-sample feature maps using averaging, maxima, stochastic selection, or multilevel and region-based aggregation. The reviewed methods differ in how they retain information and produce compact representations.
- Average Pooling: Average pooling divides the input into rectangular regions and computes each region’s average for down-sampling.
- Max-Pooling: Max pooling forwards the maximum activation within each pooling region and reduces the dimensionality of convolutional outputs.With pooling shift N < R, neighboring pooling regions can overlap.
- Mixed Pooling: Mixed pooling randomly combines average and max pooling, with λ = 0 selecting average pooling and λ = 1 selecting max pooling.The selected λ value is recorded during forward propagation and reused during backpropagation.
- Mixed Pooling: Mixed pooling was reported as superior to max and average pooling in image classification on three datasets.
- LP Pooling: LP pooling uses a parameter p to interpolate between average pooling at p = 1 and max pooling at p = ∞.Values p > 1 are examined as a trade-off between average and max pooling.
- Stochastic Pooling: Stochastic pooling samples an activation from each region using probabilities normalized from activations, retaining non-maximal activations.The method gives stronger activations higher selection probabilities and was described as limiting overfitting.
3. Novel Pooling Methods
Novel pooling methods extend conventional down-sampling by incorporating multiscale, local-structure, or learned representations to preserve useful information while improving invariance or efficiency.
- Multi-scale order-less pooling (MOP): MOP processes whole signals and local patches to combine global spatial layout with fine-grained image details.It extracts activation features at multiple scales and uses VLAD encoding to aggregate local patch features.
- Multi-scale order-less pooling (MOP): MOP concatenates original global deep activations with VLAD features from finer scales to form a new image representation.The described implementation uses a 4096-D whole-image activation, 128×128 patches, and 64×64 patches, with a 100-center codebook for VLAD pooling.
- Super-pixel Pooling: Super-pixels reduce the number of image primitives and are used in computer-vision tasks including detection, segmentation, saliency, optical flow, depth estimation, and tracking.The paper also refers to an example of super-pixel segmentation in Fig. 6.
- Super-pixel Pooling: Super-pixel pooling uses super-pixel segmentation as the pooling layout to reflect low-level image structures for semantic learning and inference.Super-pixels group image pixels according to low-level properties and provide a perceptually meaningful image tessellation.
3.3. PCA Networks
The reviewed methods use PCA-based filter learning and compact bilinear representations to improve robustness or reduce the impractical dimensionality of bilinear features.
- PCA Networks: PCANet learns multistage PCA filter banks followed by binary hashing and block histograms for indexing and pooling.The architecture is described as easily and efficiently designed and learned.
- PCA Networks: OPCA differs from PCANet by omitting hashing and local histograms in the output layer.With noise covariances as input, OPCA gains additional robustness to noise and distortions.
- PCA Networks: The baseline PCANet can incorporate OPCA's merits, potentially offering greater robustness to intraclass variability.The passage presents this as a likely benefit rather than a reported measured result.
- Compact Bilinear Pooling: Compact bilinear pooling is learned through end-to-end back-propagation and produces a low-dimensional but highly discriminative image representation.The diagram describes Tensor Sketch projection at spatial locations and summation to obtain a global compact descriptor.
- Compact Bilinear Pooling: Compact bilinear pooling addresses bilinear features' dimensionality problem, since pairwise channel interactions can produce hundreds of thousands to millions of dimensions.The method uses a sampling-based approximation to reduce feature dimensions by two orders of magnitude without a performance drop.
3.5. Lead Asymmetric Pooling (LAP)
Lead Asymmetric Pooling replaces normal pooling to address the inability of a single pooling factor to capture multiscale features, particularly in multi-lead ECG.
- Lead Asymmetric Pooling (LAP): Conventional 2-D CNN pooling downsamples feature maps using a pooling factor and produces one output for each non-overlapping sub-region.The described strategy uses a single pooling factor.
- Lead Asymmetric Pooling (LAP): A single pooling factor cannot capture multiscale features in multi-lead ECG.The paper identifies this as a limitation of normal pooling for diverse multi-lead ECG signals.
- Lead Asymmetric Pooling (LAP): Multilevel pooling can utilize multiscale features, enhance local-feature invariance, and improve CNN accuracy according to cited image-recognition studies.These properties motivate adapting multilevel pooling to multi-lead ECG.
- Lead Asymmetric Pooling (LAP): LAP replaces normal pooling and applies multiple pooling factors to multiple levels according to the level division.It is presented as an extension of the multilevel pooling strategy for managing multi-lead ECG diversity.
3.6. Edge-aware Pyramid Pooling
Edge-aware pyramid pooling preserves edge structure by integrating edge-aware feature maps, while spectral pooling reduces dimensionality through frequency-domain truncation and retains more information than max pooling.
- Edge-aware pyramid pooling preserves more edge-structure information by integrating an edge-aware feature map.
- The edge-related information assists pedestrian contour detection and motion prediction tasks.
- Compared with max pooling, spectral pooling preserves more information for the same output dimensionality through linear low-pass filtering.
- Spectral pooling applies matrix truncation with fast Fourier transformation to reduce CNN computation cost.
- Spectral pooling projects feature maps onto the Fourier basis, crops the frequency representation, and reconstructs the spatial output using inverse DFT.
- Spectral pooling permits arbitrary output map dimensionality while avoiding sharp reductions in output resolution.
3.8. Row-Wise Max-Pooling
Row-wise max pooling aggregates each input row into a maximum and supports rotation-invariant 3-D shape representations, including DeepPano, which outperforms previous methods on classification and retrieval.
- Row-wise max pooling takes the maximum value from each input row and concatenates these values into an output vector.
- The row-wise max-pooling output is unaffected by shifts of the input map, providing rotation invariance for 3-D shape representations.
- DeepPano is a rotation-invariant deep representation for 3-D shape classification and retrieval based on panoramic views.
- DeepPano outperforms previous methods by a large margin on both classification and retrieval tasks.
- The authors experimentally verified the rotation invariance of the learned representation.
3.10. Per-pixel Pyramid Pooling
Per-pixel pyramid pooling combines multiple window sizes without strides to retain coarse-to-fine information, while rank-based pooling addresses information loss from average and max pooling.
- Using one large pooling window can achieve the desired receptive field but may lose finer details.
- Multiple pooling windows with varying sizes are concatenated to create feature maps containing coarse-to-fine information.
- Per-pixel pyramid pooling performs multi-scale pooling for every pixel without strides, using a vector of pooling sizes.
- Rank-based pooling is motivated by the loss of useful information when average pooling downplays high activations and max pooling discards non-maximum activations.
- Rank-based average pooling is presented as a way to overcome information loss associated with average and max pooling.
- In rank-based average pooling, the rank threshold determines which activations are included in averaging; setting t = 1 yields max pooling.
3.12. Weighted Pooling
Weighted pooling assigns each neuron a usefulness-based weight and combines responses accordingly, allowing different proportions of local feature-map information to contribute to the output.
- Weighted pooling considers each neuron's response together with the usefulness of that response.
- For a p_w × p_h pooling window, each neuron has a response value and a corresponding weighting factor.
- The pooling result is calculated according to Eq. 9 using the weights assigned to individual responses.
- Weighted pooling captures different proportions of local information from the original feature map, leading to a better local representation.
3.13. Genetic-Based Pooling
Genetic-based pooling uses evolutionary optimization to search attention weights while reducing the model’s size and training difficulty. The method evaluates populations of weights across generations using selection, crossover, and mutation.
- Genetic pooling applies a genetic algorithm to optimize attention weights for pooling.The algorithm evolves candidate weight populations toward lower loss.
- Using genetic algorithms for pooling can make the model smaller and easier to train.
- The method randomly initializes attention-weight populations between [0,1] and trains the model for each candidate set.Error is calculated for each set of attention weights.
- The genetic pooling algorithm consists of population initialization, feed-forward evaluation, selection, crossover, mutation, and replacement.
- Each generation selects fitter individuals, applies crossover and mutation, and replaces the worst half with new children.The procedure is summarized as Genetic Pooling in Algorithm 1.
4. Conclusion
The paper situates pooling as a basic component of convolutional DNNs and reviews famous and useful pooling methods published from 1989 to 2020. It organizes the reviewed methods into popular and novel categories.
- Pooling is identified as one of the basic elements of convolutional-based DNNs.
- The paper reviews famous and useful methods for implementing pooling layers.
- The reviewed pooling methods span the period from 1989 to 2020 and are divided into popular and novel methods.