Source-linked AI summary
Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree
Chen-Yu Lee, Patrick W. Gallagher, Zhuowen Tu
TL;DR
The paper addresses the limited adaptability of conventional CNN pooling by learning pooling functions that combine max and average operations or learned filters in a tree. These generalized operations improve benchmark performance and invariance properties while remaining applicable as practical replacements with modest overhead and parameter increases.
Problem
CNN pooling is central to invariance, but conventional choices such as average and max pooling provide limited adaptation to complex and variable patterns.
Method
The paper learns pooling through mixed or gated combinations of max and average pooling and through differentiable tree-structured combinations of learned pooling filters.
Results
The proposed pooling methods boost performance across standard benchmarks and ImageNet, with state-of-the-art results on MNIST, CIFAR10, and SVHN.
Takeaways & Limitations
The methods can serve as drop-in replacements across architectures, improving performance and transformation robustness with 5% to 15% additional timing overhead and modest parameter increases.
Takeaways & Limitations
On MNIST’s very small images, the second pooling layer uses 2 × 2 pooling regions.
Abstract
from arXiv · showhide
We seek to improve deep neural networks by generalizing the pooling operations that play a central role in current architectures. We pursue a careful exploration of approaches to allow pooling to learn and to adapt to complex and variable patterns. The two primary directions lie in (1) learning a pooling function via (two strategies of) combining of max and average pooling, and (2) learning a pooling function in the form of a tree-structured fusion of pooling filters that are themselves learned. In our experiments every generalized pooling operation we explore improves performance when used in place of average or max pooling. We experimentally demonstrate that the proposed pooling operations provide a boost in invariance properties relative to conventional pooling and set the state of the art on several widely adopted benchmark datasets; they are also easy to implement, and can be applied within various deep neural network architectures. These benefits come with only a light increase in computational overhead during training and a very modest increase in the number of model parameters.
1 Introduction
The paper generalizes CNN pooling so it can learn and respond to pooled-region characteristics, using combinations of max and average pooling or tree-structured learned filters. Experiments report improved benchmark performance, state-of-the-art results on several datasets, and modest implementation costs.
- Pooling has remained largely limited to average, max, and stochastic operations despite its central role in invariance to data variation and perturbation.
- The paper introduces mixed and gated max-average pooling, respectively combining max and average pooling without or with responsiveness to pooled-region characteristics.
- Tree pooling learns pooling filters at binary-tree leaves and differentiably mixes their outputs through internal nodes to produce the root output.
- Responsive tree pooling can learn pooling filters and combine them into region-dependent behaviors within a hierarchical structure.
- 5% to 15% additional timing overhead accompanied the proposed operations, which were also differentiable and required only modest additional parameters.
- The proposed methods boosted performance across investigated standard benchmarks and ImageNet, achieving state-of-the-art results on MNIST, CIFAR10, and SVHN.
2 Related Work
Prior work includes conventional and complex pooling methods, receptive-field learning, and differentiable tree-based models. The paper distinguishes its approach by learning pooling filters and combining them differentiably within CNN-compatible training.
- Popular CNN pooling functions include max, average, and stochastic pooling, while spatial pyramid pooling mainly addresses varying image sizes rather than learning pooling functions.
- Receptive-field learning is analogous to learning pooling functions, but one cited method reported a 16.89% error rate on unaugmented CIFAR10.
- Standard decision trees are nondifferentiable, whereas the paper ensures its tree-based pooling operation is differentiable for CNN backpropagation.
- Auto-encoder trees learn encoding and decoding methods for generative models, unlike this paper’s pooling-focused method.
- A cited supervised decision-forest approach uses separately trained perceptrons as splitting nodes and does not learn pooling filters.
3 Generalizing Pooling Operations
The paper generalizes CNN pooling by learning combinations of conventional max and average pooling, as well as tree-structured combinations of learned pooling filters. These operations can respond to pooled-region characteristics and improve benchmark performance with modest parameter increases.
- Overview: Pooling is generalized through learned combinations of max and average pooling, plus tree pooling that learns both pooling filters and their combinations.Mixed pooling uses fixed max and average operations, while tree pooling learns filters associated with tree leaves and combines their outputs.
- Mixed max-average pooling: Mixed max-average pooling learns a fixed mixing proportion between max and average pooling.For the one-per-layer option, the operation is fmix(x) = a_l·fmax(x) + (1 − a_l)·favg(x), with a_l in [0, 1].
- Gated max-average pooling: Gated max-average pooling uses a learned gating mask to make the max-average mixing proportion responsive to the pooled region.The inner product of the gating mask and region is passed through a sigmoid to determine the mixing proportion.
- Empirical comparisons: Across datasets, responsive gated pooling generally outperforms nonresponsive mixed pooling despite using far fewer parameters, with SVHN as the reported exception.The gated strategy uses 18 extra parameters in the cited 3 × 3, two-pooling-layer example, versus more than 40,000 for the compared mixed configuration.
- Tree pooling: Tree pooling combines learned pooling filters in a pre-specified binary tree, with each leaf associated with a learned filter.Responsive node mixing allows the operation to display different behaviors according to pooled-region characteristics.
- Empirical comparisons: Tree pooling benefits from additional levels on all datasets except MNIST, while combining tree pooling first with gated max-average pooling second provides further benefit.The proposed pooling method also achieves 7.62% error with only 72 extra parameters, compared with at least 9.08% for the explored deeper convolutional alternatives.
4 Quick Performance Overview
Experiments examine invariance, performance, and computational cost for generalized pooling operations across benchmark settings. The proposed methods improve transformed-image accuracy and benchmark results with modest parameter and timing overhead.
- Invariance properties: CIFAR10 accuracy improved across three transformation types when gated max-avg or two-level tree pooling replaced max or average pooling, except under extreme downscaling.The comparison used one pooling parameter per layer and evaluated transformed CIFAR10 test images.
- Performance and overhead: 72 additional parameters reduced CIFAR10 test error from 9.10% to 7.62% for the tree+gated max-avg configuration.The baseline network had 1.8M parameters.
- Performance and overhead: 6% relative reduction in top-5 single-view AlexNet test error required only 45 additional parameters.The comparison replaced AlexNet maxpool layers and used a baseline exceeding 50M parameters.
- Invariance properties: The invariance experiment evaluates four networks—max, average, gated max-avg, and two-level tree pooling—on transformed CIFAR10 test sets.Each network was trained on CIFAR10 training data before evaluation under varied transformations.
- Performance and overhead: 5% to 15% additional computation time was incurred by the proposed pooling operations during forward and backward processing.The timing experiment measured average forward+backward time per CIFAR10 image.
5 Experiments
The experiments evaluate learned max-average and tree pooling across five benchmark datasets and ImageNet, using matched network structures and controlled pooling substitutions. Proposed configurations improve classification results while adding few parameters, including state-of-the-art outcomes on several datasets.
- Evaluation setup: Five benchmark datasets—MNIST, CIFAR10, CIFAR100, SVHN, and ImageNet—were evaluated with data and preparation matched to prior work.The proposed max-average and tree pooling approaches were tested under controlled experimental conditions.
- Evaluation setup: All results use the same network structure and hyperparameters, differing only in the chosen pooling function.The baseline uses conventional max pooling; mixed, gated, tree, and tree-plus-max-average configurations replace pooling operations in specified layers.
- Classification results: 0.3% improvement over the baseline model was obtained by learning a single CIFAR10 pooling filter, with further improvement when combining learned pooling filters.These results support progressively learning both pooling filters and their combinations.
- Parameter efficiency: 63 tree-pooling parameters plus 9 gating parameters achieved 7.62% CIFAR10 error, compared with 9.08% for All-CNN using nearly 500,000 pooling-like parameters.The comparison concerns the reported pooling-like layers and uses a three-level tree followed by gated max-average pooling.
- Classification results: 0.30%, 8.02%, 6.61%, 33.27%, and 1.64% test error were obtained on MNIST, CIFAR10, CIFAR10+, CIFAR100, and SVHN, respectively, with mixed max-average pooling.The SVHN result was reported as a new state of the art, while performance varied relative to other max-average configurations across datasets.
6 Observations from Experiments
Across experiments, generalized pooling consistently improves performance and remains effective alongside multiple training enhancements and network architectures. A fixed tree-plus-max-average configuration reaches state-of-the-art performance on several benchmark settings.
- Performance across settings: Every proposed pooling operation boosted performance in the reported experiments.The improvements were observed across the evaluated experimental settings rather than only one pooling variant.
- Performance across settings: A fixed tree+max-avg configuration achieved state-of-the-art performance on MNIST, CIFAR10 with and without augmentation, and SVHN.The same configuration also produced boosts with data augmentation, multi-view predictions, batch normalization, and several architectures.
- Performance across settings: Performance boosts accompanied NiN-style, DSN-style, AlexNet, and GoogLeNet architectures, including the >50M-parameter AlexNet and 22-layer GoogLeNet.The reported gains also occurred alongside data augmentation, multi-view predictions, and batch normalization.
A1 Supplementary Materials
The supplementary material visualizes internal pooling representations on CIFAR10 with t-SNE across pooling methods and two pooling layers. The visualization compares class separation after successive pooling operations.
- Visualization of network internal representations: Certain classes, including 0, 2, and 9, are more separated with the proposed pooling methods in the plotted embeddings.The figure examines activations immediately after the first and second pooling operations.
- Visualization of network internal representations: Figure A1 compares t-SNE embeddings for average, max, gated max-avg, and two-level tree pooling on CIFAR10 test responses.Columns correspond to pooling methods, while rows correspond to the first and second pooling layers.
- Visualization of network internal representations: Second-pooling-layer activations are generally more separable than first-pooling-layer activations.This comparison concerns the embeddings shown in the second and first rows, respectively.
- Experimental conditions: The supplementary table explicitly lists the convolutional and pooling-like layer configurations used in the main experimental comparisons.ReLUs and deep supervision are omitted from the display for clarity, although the configurations incorporate them.