PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaPool: Exponential Adaptive Pooling for Information-Retaining Downsampling

Alexandros Stergiou, Ronald Poppe

arXiv:2111.00772v3cs.CV

TL;DR

Pooling must reduce resolution efficiently without discarding informative detail, and conventional pooling offers limited reversibility for upsampling. The paper introduces AdaPool, an adaptive fusion of exponential Dice-Sørensen and exponential maximum pooling, plus AdaUnPool and Inter4K. Across varied tasks and backbones, AdaPool systematically outperforms other methods while keeping computational latency and memory use modest.

  • Problem

    Pooling reduces computation and increases receptive fields but can lose structural detail, while conventional pooling generally cannot be reversed for information-aware upsampling.

  • Method

    AdaPool learns a regional fusion of exponential maximum and exponential Dice-Sørensen pooling, with AdaUnPool reusing the learned formulation for upsampling.

  • Results

    Across image and video classification, similarity, detection, super-resolution, and interpolation tasks, AdaPool systematically outperforms other methods with modest latency and memory use.

  • Takeaways & Limitations

    AdaPool is presented as a practical alternative to popular pooling operators, while AdaUnPool extends the approach to upsampling tasks.

Abstract

from arXiv · show

Pooling layers are essential building blocks of convolutional neural networks (CNNs), to reduce computational overhead and increase the receptive fields of proceeding convolutional operations. Their goal is to produce downsampled volumes that closely resemble the input volume while, ideally, also being computationally and memory efficient. Meeting both these requirements remains a challenge. To this end, we propose an adaptive and exponentially weighted pooling method: adaPool. Our method learns a regional-specific fusion of two sets of pooling kernels that are based on the exponent of the Dice-Sorensen coefficient and the exponential maximum, respectively. AdaPool improves the preservation of detail on a range of tasks including image and video classification and object detection. A key property of adaPool is its bidirectional nature. In contrast to common pooling methods, the learned weights can also be used to upsample activation maps. We term this method adaUnPool. We evaluate adaUnPool on image and video super-resolution and frame interpolation. For benchmarking, we introduce Inter4K, a novel high-quality, high frame-rate video dataset. Our experiments demonstrate that adaPool systematically achieves better results across tasks and backbones, while introducing a minor additional computational and memory overhead.

I. INTRODUCTION

Pooling reduces computation and expands receptive fields but can discard structural detail. AdaPool addresses this trade-off by adaptively combining exponential maximum and exponential Dice-Sørensen weighting, while extending the same formulation to upsampling.

  • Pooling lowers spatial resolution and computational overhead but can lose contrast, texture, and other structural detail.
  • Existing maximum and average pooling methods are efficient but leave room for improving information retention, while trainable alternatives can be less efficient or require task-specific parameters.
  • AdaPool learns a regional fusion of exponential maximum and exponential Dice-Sørensen weighting to balance informative detail with local image structure.
  • AdaUnPool reuses AdaPool’s learned weights as prior local knowledge for upsampling, addressing the limited reversibility of conventional pooling.
  • The method extends inverse-distance pooling with exponentiated Dice-Sørensen similarity, providing differentiability and minimum gradients across locations.
  • The paper introduces Inter4K, a 1,000-video high-frame-rate 4K dataset, and evaluates AdaPool across classification, recognition, detection, super-resolution, and interpolation tasks.

II. RELATED WORK

CNN pooling reduces spatial resolution and computation but can lose structural detail. Prior methods improve retention or learn weighting, while adaPool adaptively combines exponential weighting strategies and supports inverse upsampling.

  • Motivation: Pooling reduces spatial resolution and subsequent computational requirements, but can lose informative structural features such as contrast and texture.This information loss makes detail retention difficult in CNN-based image and video processing.
  • Upsampling limitations: Most pooling methods cannot be inverted for upsampling, while maximum-pooling inversion produces inherently sparse outputs.Interpolation and transposed or de-convolutional methods approximate higher-resolution features rather than reconstructing them.
  • Proposed approach: AdaPool adaptively fuses exponential weighting based on activation softmax weighting and Dice-Sørensen similarity to retain input detail.Its weighting approach is intended to avoid averaging away high-frequency patterns and improve information retention.
  • Reported effects: AdaPool calculates gradients for each kernel vector and reduces the likelihood of vanishing activation trends during backpropagation.The paper reports clearer and more recognizable letters and numbers in its detail-preservation example.

B. Smooth approximated maximum pooling

Exponential maximum pooling provides a smooth approximation to maximum pooling by weighting activations according to their magnitudes. It emphasizes larger activations while retaining nonzero influence from lower activations and multiple feature values.

  • Definition: Exponential maximum pooling is a smooth approximation of SoftPool that uses the natural exponent to weight activations.The method produces normalized results through a probability distribution proportional to activation values within the kernel region.
  • Weighting behavior: Higher-valued activations receive greater influence, while the lowest activations retain a minimum weight.This differs from selecting only the maximum activation, which can discard information from the remaining activations.
  • Output: The eMPool output is computed by summing all weighted activations within the kernel region.Its weighting emphasizes dominant activations without reducing the operation to maximum selection alone.

C. AdaPool: Adaptive exponential pooling

AdaPool combines two exponential pooling strategies with a regionally learned weight mask, while adaUnPool reuses pooling-derived information for upsampling.

  • eDSCWPool weights activations by similarity to the regional mean, whereas eMPool weights them according to activation values.
  • AdaPool learns β to fuse smooth approximations of average and maximum pooling within each kernel region.The β mask matches the downsampled volume and is learned regionally.
  • AdaUnPool uses adaPool’s differentiable, minimum-weight assignments as prior information for inflating pooled volumes during upsampling.The operation expands activations from H′×W′ to H×W.
  • Inter4K visualizations depict video categories and continental locations, with darker map colors indicating more videos.

IV. THE INTER4K VIDEO DATASET

Inter4K is a standardized high-resolution, high-frame-rate video dataset designed to benchmark super-resolution and frame interpolation across multiple scales and frame rates.

  • Inter4K contains 1,000 4K, 60-fps video clips sourced from YouTube for benchmarking upsampling methods.
  • The dataset standardizes resolutions from UHD/4K through nHD/360p and frame rates from 60 to 15 fps.These settings support ×2, ×3, and ×4 scaling experiments.
  • Inter4K is divided into 800 training, 100 validation, and 100 testing videos, each five seconds long.
  • The videos vary in equipment, lighting, camera motion, actions, and objects, and are organized into six broad categories.
  • The experiments use Inter4K for frame upsampling and interpolation alongside image and video datasets for related evaluations.The broader evaluation also covers downsampling, classification, detection, and image super-resolution.

B. Downsampling similarity

AdaPool preserves downsampled image similarity better than alternative methods while maintaining reasonable inference costs, and replacing pooling layers improves classification accuracy across CNN backbones.

  • AdaPool consistently achieves the best overall SSIM, PSNR, and LPIPS performance across kernel sizes and high-resolution image datasets.The evaluations average results over DIV2K, Urban100, Manga109, and Flicker2K.
  • Exponential methods eMPool and eDSCWPool outperform nontrainable and stochastic pooling methods on downsampling similarity.
  • AdaPool implementations achieve reasonable CUDA inference times despite additional computations compared with simpler pooling methods.The reported overheads include forward and backward passes on CPU and GPU.
  • AdaPool replacements improve classification accuracy across tested CNN architectures on ImageNet1K.
  • 2.19% top-1 and 1.33% top-5 average gains are reported for ResNet models using adaPool.The improvements are measured against the original ResNet models.
  • 2.37% top-1 and 1.17% top-5 average gains are reported for ResNeXt models using adaPool.

E. Comparison with alternative pooling methods

Across alternative pooling comparisons, adaPool outperforms learnable and stochastic methods, while exponential pooling also improves object-detection accuracy.

  • AdaPool outperforms other learnable and stochastic pooling methods across the tested CNN architectures.
  • 1.61–2.78% improvements are observed for InceptionV1, while DenseNet-121 gains range from 0.65–2.64% over other methods.
  • AdaPool increases MS COCO object-detection average precision by 2.40% over the original models.eMPool and eDSCWPool produce smaller average precision improvements of 1.00% and 0.86%, respectively.
  • The object-detection improvements also appear for AP50 and AP75.

G. Video classification performance

AdaPool improves spatiotemporal CNN classification and supports bidirectional downsampling and upsampling for related video and image restoration tasks. Across reported video benchmarks, it delivers consistent gains with negligible computational overhead.

  • Video classification performance: AdaPool extends pooling to spatiotemporal kernels by including the temporal dimension in each kernel region.The method otherwise keeps the same formulation used for spatial pooling.
  • Video classification performance: 87.83% and 64.67% top-1 accuracy were achieved by MTNetL with adaPool on HACS and Kinetics-700, improving over original pooling by 1.21% and 1.36%.The reported increase adds only +0.2 GFLOPs.
  • Video classification performance: AdaPool increased MTNetL accuracy by 1.22% on UCF-101, including under near-saturated performance.The authors characterize the gain as modest but important.
  • Image and video restoration: Replacing pooling and interpolation with ada(Un)Pool improved image super-resolution and frame interpolation across the tested converted networks.CDFI with adaPool and adaUnPool achieved state-of-the-art results on Vimeo90K, Middlebury, and Inter4K.
  • Image and video restoration: CDFI with ada(Un)Pool showed only slight performance degradation for high-resolution, high-frame-rate conversions on Inter4K.The combined evaluation covered frame super-resolution and interpolation.

VI. ABLATION STUDIES

Ablations show that trainable adaptive fusion consistently outperforms fixed fusion, while replacing pooling at different depths yields systematic accuracy gains. The adaptive combination of approximated average and maximum pooling also compares favorably with alternatives.

  • Fusion-weight ablations: Trainable β masks outperform the best constant values across all three tested networks.The top-1 difference grows from 0.44% in ResNet-18 to 1.18% in InceptionV3.
  • Layer-wise ablation: Each additional InceptionV3 pooling replacement increases top-1 accuracy by an average of 0.56%.The largest gains occur at pool1 (+0.89%) and mixed7b−d (+0.80%).
  • Layer-wise ablation: AdaPool improves accuracy regardless of network depth and channel count.The layer-wise results attribute systematic gains to adaptive weighting.
  • Fusion comparisons: The smooth approximated average–maximum combination performs favorably across tested average- and maximum-based fusion strategies.Adaptive parameter masks further improve performance over simpler fusion choices.

D. Comparisons to attention-based downsampling

AdaPool offers a low-cost alternative to attention-based downsampling while preserving descriptive features in visual representations. Qualitative analyses associate its use with more focused saliency and somewhat closer embeddings for similar examples.

  • Attention-based comparisons: AdaPool requires only +1.5 MFLOPs and 4.2K parameters, making it substantially more efficient than the tested attention-based methods.It performs favorably against SE- and CBAM-based pooling, with a small performance decrease versus some MSA variants.
  • Saliency visualizations: AdaPool-based saliency maps better fit regions containing multiple objects and descriptive class features.Examples include sails for pirate ships and the racket and ball for tennis balls.
  • Feature embeddings: AdaPool produces somewhat closer embeddings for similar examples and clearer distinctions among bell-pepper colors and object counts.These comparisons use t-SNE visualizations of original and adaPool-replaced InceptionV3 features.
  • Method rationale: AdaPool’s adaptive exponential weighting combines exponential maximum and exponential average pooling through regional parameterized fusion.The formulation is intended to preserve informative features while downsampling.
  • Bidirectional operation: The learned adaPool weights can be reused to invert pooling for upsampling through adaUnPool.This bidirectional property extends the method beyond downsampling.

APPENDIX A

The appendix compares distance- and similarity-based pooling alternatives and motivates similarity weighting for high-dimensional activation regions. It reports that DSC-based exponential weighting performs strongly, while distance choices differ little.

  • Inverse Distance Weighting pooling: IDWPool assigns larger normalized weights to activation vectors closer to the regional mean and reduces the effect of outliers.The reported experiments use Euclidean distance between the mean and individual activations.
  • Inverse Distance Weighting pooling: Distance-based weighting is sub-optimal in multidimensional spaces because unbounded, outlier-sensitive distances can produce near-zero weights and pixel artifacts.The limitation is attributed to channel-wise distance variation.
  • Distance-function comparison: Huber-based IDW pooling improved top-1 accuracy by +(0.10–0.19)% over L1, L2, and Chebyshev weighting, while differences among distances were otherwise negligible.A slight reduction was observed for Gower weighting.
  • Similarity-based alternatives: Similarity measures improve more over the baseline than distance approaches for the tested ResNet-18 pooling alternatives.The authors attribute this to sparse per-input volumes and large distances in high-dimensional kernel regions.

D. Ablations over β parameterization alternatives

The ablation compares shared, location-wise, and channel-wise β parameterizations to test whether adaPool’s gains come from information retention rather than added parameters. Results indicate that the proposed non-channel-wise parameterization performs better while maintaining implementation efficiency close to standard pooling methods.

  • β parameterization alternatives: AdaPool is evaluated with shared, location-wise, and channel-wise β masks to separate information-retention gains from effects of additional parameters.The tested sizes are |β| = 1, |β| = H′×W′, and |β| = H′×W′×C.
  • β parameterization alternatives: The results suggest that adaPool’s improvements are not solely dependent on including additional parameters.This conclusion follows from comparing parameterizations with different β sizes.
  • β parameterization alternatives: The channel-wise β adds 1.01% in ResNet-50 and 1.28% in DenseNet-121, but underperforms the non-channel-wise alternatives.The comparison is reported on ImageNet1K for both backbones.
  • β parameterization alternatives: The authors suggest that the pooling approach is better suited to data with larger channel and feature dependencies.
  • Computational considerations: CUDA implementations of exponential maximum and average pooling achieve inference times close to native average and maximum pooling through parallelization and tiling.The implementation also supports fused multiply-add operations on CUDA-enabled devices.
Loading 2111.00772v3…