Source-linked AI summary
Detail-Preserving Pooling in Deep Networks
Faraz Saeedan, Nicolas Weber, Michael Goesele, Stefan Roth
TL;DR
Conventional pooling must downscale activations despite trade-offs among detail preservation, gradient flow, and the network–dataset-specific choice of pooling strategy. This paper proposes learnable detail-preserving pooling, which emphasizes local structural detail. The authors report consistent advantages across tested pooling settings, including improved ImageNet results, while noting a formulation constraint on learning p.
Problem
Standard pooling can discard important local detail or disrupt gradient flow, and the best strategy depends on the network and dataset.
Source: §1 · p. 1Method
DPP uses learnable inverse-bilateral rewards to pool each feature map, with symmetric or asymmetric variants, and can also be combined with stochastic sampling.
Source: §4 · pp. 3–4Results
Takeaways & Limitations
The authors conclude that DPP is broadly applicable across a wide range of network architectures and datasets, with only minor computational overhead.
Source: §5 · p. 5, §7 · p. 8Learning the p value requires a modified formulation because of possible exploding gradients.
Source: §6.1 · p. 7
Abstract
from arXiv · showhide
Most convolutional neural networks use some method for gradually downscaling the size of the hidden layers. This is commonly referred to as pooling, and is applied to reduce the number of parameters, improve invariance to certain distortions, and increase the receptive field size. Since pooling by nature is a lossy process, it is crucial that each such layer maintains the portion of the activations that is most important for the network's discriminability. Yet, simple maximization or averaging over blocks, max or average pooling, or plain downsampling in the form of strided convolutions are the standard. In this paper, we aim to leverage recent results on image downscaling for the purposes of deep learning. Inspired by the human visual system, which focuses on local spatial changes, we propose detail-preserving pooling (DPP), an adaptive pooling method that magnifies spatial changes and preserves important structural detail. Importantly, its parameters can be learned jointly with the rest of the network. We analyze some of its theoretical properties and show its empirical benefits on several datasets and networks, where DPP consistently outperforms previous pooling approaches.
1. Introduction
Standard pooling reduces cost and expands receptive fields, but can discard detail or disrupt gradient flow, while the best strategy varies by network and dataset. The paper introduces learnable detail-preserving pooling and reports gains across CIFAR10 settings and ImageNet classification.
1. Introduction
- Strided convolutions ignore activation significance, average pooling attenuates local contributions uniformly, and max pooling selects one node, disrupting some gradient branches.These approaches can respectively cause aliasing, loss of detail, or implausible downscaled images.Source: §1 · p. 1
- The best standard pooling choice depends on the network–dataset combination, motivating exhaustive testing and leaving generic nonlinear pooling benefits uncertain.Prior nonlinear approaches may require brute-force parameter search, with benefits for deep networks or large-scale tasks remaining unclear.Source: §1 · pp. 1–2
- DPP learns from a continuum spanning average to max or extremum pooling, can form nonlinear combinations, is differentiable, and can combine with stochastic regularization.Its parameters are learned jointly with the network.Source: §1 · p. 2
- On CIFAR10, DPP performs at least as well as the best standard pooling across network types and beats other recent pooling methods in the tested settings; on ImageNet it improves results.ResNet-101 with DPP outperforms plain ResNet-152 despite having significantly fewer layers and parameters.Source: §1 · p. 2
2. Related Work
Prior pooling methods address invariance, receptive fields, and downsampling in different ways, while many learnable approaches combine existing pooling operations. DPP is presented as a trainable, differentiable alternative that includes major pooling techniques as limit cases.
2. Related Work
- Max pooling, average pooling, and strided convolutions are common CNN choices; extremum pooling selects the activation that deviates most from its neighborhood mean, whether maximum or minimum.Strided convolutions provide efficient but data-independent sparse regular downsampling.Source: §2 · p. 2
- DPP nonlinearly bridges average and max pooling and learns the function shape for each feature map from training data and its activations.This follows the observation that optimal pooling for a feature may lie between average and max, particularly given max pooling's discriminability benefits for low-activation-probability features.Source: §2 · p. 2
- Other approaches alter receptive fields or pooling ratios, use higher-order or inter-channel pooling, or downsample in spectral space with low-pass filtering.The cited spectral method smooths inputs rather than preserving details concentrated in higher frequencies.Source: §2 · p. 2
- Learnable alternatives include hard switching between max and average, spatial Lp norms, learned trees, and stochastic node selection.S3pool and stochastic pooling combine pooling with regularization, making their individual benefits difficult to assess.Source: §2 · p. 2
- DPP targets trainable, fully differentiable downscaling inside networks and includes major pooling techniques as special limit cases.Other pooling applications include scaling with input size and global averaging before classification.Source: §2 · p. 2
- DPP omits DPID's Gaussian filter, while Full-DPP replaces the box filter with a learned 2D filter.These are the two structural differences identified in the diagram.Source: Figure 2 · p. 3
3. Detail-Preserving Image Downscaling
Detail-preserving image downscaling uses inverse bilateral weighting to emphasize pixel differences from a linearly downscaled image. The reward strength controls how strongly local detail influences the output.
3. Detail-Preserving Image Downscaling
- DPID emphasizes small image details because they can carry more information than larger areas of similar color, using an inverse bilateral filter to reward rather than penalize differences.The method is motivated by preserving details for a faithful visual impression.Source: §3 · p. 3
- DPID forms a linearly downscaled image from box filtering and downsampling, then smooths it with an approximate 2D Gaussian filter.The neighborhood size corresponds to the downscaling ratio, yielding one output pixel per input neighborhood.Source: §3 · p. 3
- The output is a weighted average that gives greater influence to input pixels farther in intensity from the linear downscale, rewarding both darker and lighter deviations.This allows edges and corners to have greater influence.Source: §3 · p. 3
- The inverse bilateral reward functions can be symmetric, favoring larger absolute values, or asymmetric, favoring larger positive values; as λ approaches zero, weights become uniform and averaging results.Smaller λ reduces the reward function's dynamic range.Source: Figure 3 · p. 3
4. Detail-Preserving Pooling
Detail-preserving pooling (DPP) uses learned, inverse-bilateral weights to preserve detail when downscaling feature maps. Its symmetric and asymmetric variants, plus full, lite, and stochastic forms, offer different ways to apply this idea.
Detail-Preserving Pooling
- DPP computes a spatially weighted average per feature map, using a learned reward function to emphasize details in local activation neighborhoods.The paper defines symmetric rewards that enhance all details and asymmetric rewards that enhance details with higher-than-average activation.Source: §4 · p. 3
Inverse bilateral weights
- The learned reward exponent λ shapes the nonlinear weights, while bias α keeps inputs from uniform regions from being entirely eliminated.The symmetric reward uses a differentiable generalized Charbonnier penalty; the asymmetric reward favors activations at or above the local average.Source: §4 · pp. 3–4
Linear downscaling
- Full-DPP uses a learned, non-normalized 2D filter, typically over 3×3 regions; its pooling ratio is set by the following downsampling stride.The filtering regions can differ in size from the inverse-bilateral neighborhoods.Source: §4 · p. 4
- Lite-DPP replaces the learned filter with a box filter and uses 2×2 regions, which the authors found sufficient because 3×3 regions showed no improvement.Source: §4 · p. 4
Learning and differentiability
- DPP is fully differentiable, allowing the network to learn α and λ and, optionally, the linear-downscaling filter parameters.The authors initialize inverse-bilateral parameters to 1 and filter coefficients to 0, then add small zero-mean Gaussian perturbations.Source: §4 · p. 4
Stochastic variants
- S3DPP applies stride-1 regular DPP before uniformly sampling rows and columns, making node importance depend on learned λ and α.The paper says this produces controlled importance sampling.Source: §4 · pp. 4–5
5. Analysis and Discussion
DPP can reproduce familiar pooling behaviors under particular parameter settings, while adding modest parameter and computation overhead. Experiments report benefits in deeper ResNet layers and improved translation invariance over max pooling, with a trade-off across translation magnitudes.
5. Analysis and Discussion
- At λ = 0, DPP is equivalent to average pooling; as λ approaches infinity, symmetric DPP becomes extremum pooling and asymmetric DPP becomes max pooling.When extrema are tied, symmetric DPP averages them in the large-λ limit.Source: §5 · p. 5
- Full-DPP adds 43k parameters to ResNet and 17.7k to VGG-16, while Lite-DPP adds even fewer parameters.The reported additions are 0.172% and 0.098% for ResNet, and 0.013% for VGG-16.Source: §5 · p. 5
- DPP slows VGG-16 on ImageNet by about 20% and ResNet-101 by about 5%; its overhead depends on pooling-layer count, not convolution-layer count.The inverse bilateral weight calculation takes 45 ms for a forward and backward pass on a mini-batch of 128 images under the specified setup.Source: §5 · p. 5
- Replacing only ResNet’s second, deeper downsampling step with DPP improves CIFAR10 accuracy, and replacing both steps improves results further.The comparison evaluates ResNet downsampling configurations on CIFAR10.Source: Table 1 · p. 5, §5 · p. 6
- DPP is more translation-invariant than max pooling and outperforms average pooling for horizontal shifts up to 4 pixels.The comparison uses VGG on CIFAR10; 4 pixels is 12.5% of the image size.Source: §5 · p. 6, Figure 5 · p. 6
6. Experiments
Experiments across CIFAR10 and ImageNet show that DPP improves on original pooling choices, with consistent benefits across tested architectures. On CIFAR10, DPP also performs strongly against advanced and stochastic pooling methods.
6. Experiments
- Experiments compare learned Lite-DPP and Full-DPP variants with asymmetric and symmetric rewards, and include stochastic S3DPP in some settings.DPP parameters are learned from data and are unique to each feature channel.Source: §6 · p. 6
6.1. Detailed analysis on CIFAR10
- On CIFAR10, DPP yields the lowest error among deterministic pooling methods for all three tested networks and reaches 5.97% error with ResNet-110.The table compares standard, advanced, and stochastic pooling methods across VGG, NIN, and ResNet; DPP also ranks best among the stochastic methods.Source: Table 2 · p. 7, §6.1 · p. 7
- No standard pooling choice is consistently best across the CIFAR10 networks, and the best choice can differ from each network’s original pooling.Mixed and gated pooling show inconsistent results across VGG, NIN, and ResNet.Source: §6.1 · p. 7
- DPP variants also perform strongly among stochastic methods, while excessive stochasticity can limit accuracy.For ResNet, the stochastic DPP variant yields 5.42% error; S3DPP makes lean DenseNet-BC competitive with the much larger standard DenseNet.Source: §6.1 · p. 7, Table 3 · p. 7
6.2. Image classification on ImageNet
- On ImageNet validation, adding DPP lowers error by 0.8% for ResNet-50 and 0.6 percentage points for ResNet-101; DPP ResNet-101 surpasses standard ResNet-152 with 5% versus 50% slowdown over ResNet-101.Source: Table 4 · p. 8, §6.2 · p. 8
7. Conclusion
The paper presents DPP, a learnable pooling layer based on inverse bilateral filters that preserves structural detail during downscaling. Across tested architectures and datasets, it reports consistent gains over standard and selected advanced pooling methods, with further gains when combined with stochastic pooling.
7. Conclusion
- DPP uses inverse bilateral filters and learnable parameters to control how much structural detail downscaling preserves.Theoretical analysis shows it can behave like max/extremum pooling, average pooling, or intermediate nonlinear functions, with minor computational overhead.Source: §7 · p. 8
- Across a wide range of tested architectures and datasets, DPP performs consistently better than the best standard pooling layer and selected advanced pooling methods.For ResNet on ImageNet, DPP lowers error even below that of much deeper models.Source: §7 · p. 8
- Combining DPP with stochastic pooling yields further accuracy gains, as the paper describes detail preservation and regularization as complementary.Source: §7 · p. 8
A. Learned Pooling Parameters
Learned parameters in symmetric Lite-DPP show different pooling tendencies across VGG layers on CIFAR10. Earlier layers lean toward extremum pooling, while deeper layers become more average-like.
A. Learned Pooling Parameters
- In symmetric Lite-DPP, earlier VGG layers learn smaller reward bias values and an extremum-like tendency, while deeper layers become more average-like.The overall trend resembles the asymmetric version, but early symmetric layers learn a smaller reward exponent, indicating somewhat less detail preservation with extremum pooling than in the asymmetric case.Source: A. Learned Pooling Parameters · p. 10, Figure 7 · p. 10
- Figure 7 plots learned λ and α across VGG pooling layers; higher λ indicates more extremum-like behavior and higher α indicates more averaging.Source: Figure 7 · p. 10
B. Network Details
The network details describe the CIFAR10 VGG-like and NIN-like architectures and the pooling-layer substitutions used in experiments.
B. Network Details
- The NIN-like network adds a fully connected layer after global average pooling to help avoid local minima.Source: B. Network Details · p. 10
- Pooling methods are compared by swapping pooling layers; strided-convolution comparisons instead set the preceding convolution strides to two.Source: B. Network Details · p. 10
- The CIFAR10 VGG-like and NIN-like network configurations are listed in Table 5, with batch normalization and ReLU following each convolution.Source: Table 5 · p. 10, B. Network Details · p. 10