PaperlayerView exact revision on arXiv

Source-linked AI summary

LIP: Local Importance-based Pooling

Ziteng Gao, Limin Wang, Gangshan Wu

arXiv:1908.04156v3cs.CV

TL;DR

Spatial downsampling supports larger receptive fields and lower memory use, but can lose discriminative details, particularly for tiny objects. The paper analyzes pooling through local importance and proposes LIP, which learns adaptive importance weights from inputs. LIP improves ImageNet results across architectures and COCO detection, with reported state-of-the-art small-object performance under single-scale testing.

  • Problem

    Existing downsampling strategies can dilute or miss discriminative details, especially those of tiny objects.

    Source: §1 · p. 1

  • Method

    The paper analyzes pooling with a local-importance framework and proposes LIP, which learns adaptive importance weights from input features.

    Source: §1 · pp. 1–2

  • Results

    LIP improves ImageNet classification across architectures and COCO detection with one- and two-stage detectors, achieving state-of-the-art small-object performance under single-scale testing.

    Source: §1 · p. 2

  • Takeaways & Limitations

    LIP-ResNet backbones improve detection performance, especially on small objects, within standard detection frameworks.

    Source: §5 · p. 8

Abstract

from arXiv · show

Spatial downsampling layers are favored in convolutional neural networks (CNNs) to downscale feature maps for larger receptive fields and less memory consumption. However, for discriminative tasks, there is a possibility that these layers lose the discriminative details due to improper pooling strategies, which could hinder the learning process and eventually result in suboptimal models. In this paper, we present a unified framework over the existing downsampling layers (e.g., average pooling, max pooling, and strided convolution) from a local importance view. In this framework, we analyze the issues of these widely-used pooling layers and figure out the criteria for designing an effective downsampling layer. According to this analysis, we propose a conceptually simple, general, and effective pooling layer based on local importance modeling, termed as {\em Local Importance-based Pooling} (LIP). LIP can automatically enhance discriminative features during the downsampling procedure by learning adaptive importance weights based on inputs. Experiment results show that LIP consistently yields notable gains with different depths and different architectures on ImageNet classification. In the challenging MS COCO dataset, detectors with our LIP-ResNets as backbones obtain a consistent improvement ($\ge 1.4\%$) over the vanilla ResNets, and especially achieve the current state-of-the-art performance in detecting small objects under the single-scale testing scheme.

1. Introduction

Spatial downsampling reduces feature-map size and expands receptive fields, but pooling can lose discriminative details, especially those of tiny objects. The paper analyzes pooling through local importance and proposes LIP, which learns input-dependent importance weights to preserve useful features; experiments report gains on ImageNet and COCO.

1. Introduction

  • Pooling layers reduce feature-map size for larger receptive fields and lower memory consumption, but use different strategies to aggregate each sliding window.Source: §1 · p. 1
  • Improper downsampling can dilute tiny-object details with clutter activations or fail to sample them, hindering recognition and detection.Source: §1 · p. 1
  • LIP learns input-dependent local importance weights to adaptively preserve discriminative features during downsampling.The framework views pooling as aggregating features according to local importance; LIP can mimic average pooling, max pooling, and detail-preserving pooling [33].Source: §1 · p. 2
  • LIP outperforms baselines on ImageNet across architectures, while LIP-ResNet backbones improve one- and two-stage COCO detectors and achieve state-of-the-art small-object detection under single-scale testing.Source: §1 · p. 2

2. Related Work

Prior pooling research includes methods for fixed-size outputs, spatial transformation, and combinations or generalizations of average and max pooling. This paper frames pooling through local importance, while distinguishing LIP from hand-crafted detail-preserving pooling and other attention approaches.

2. Related Work

  • CNNs use average and max pooling in VGG, Inception, and DenseNet, while ResNet uses strided convolutions for downsampling.Source: §2 · p. 2
  • Earlier approaches include pooling arbitrary-sized feature maps to fixed sizes, which the paper excludes because those methods target specific architectures.Source: §2 · p. 2
  • Fractional pooling and S3pool change spatial transformation, while mixed, hybrid, and Lp pooling combine or generalize average and max pooling.Lp pooling uses a learned p to control a continuum between max and average pooling.Source: §2 · p. 2
  • Detail-preserving pooling preserves details and discards redundant features, but its deviation-from-statistics criterion is hand-crafted and potentially suboptimal.Source: §2 · p. 2
  • The paper analyzes pooling through local importance, a view it says had not been investigated previously, and reports that LIP outperforms hand-crafted pooling layers.Source: §2 · p. 2
  • LIP can be viewed as local attention with locally normalized softmax weights, using convolutional logits rather than key-query schemes.Source: §2 · p. 2

3. Local Importance Modeling

The section frames downsampling as locally weighted aggregation, then uses that view to identify shortcomings in existing pooling methods and motivate LIP. LIP learns input-dependent importance weights, and the paper describes how it is incorporated into ResNet and DenseNet.

3. Local Importance Modeling

  • The section presents local importance modeling as a framework for analyzing existing downsampling methods and deriving criteria for pooling.It seeks adaptive importance that favors discriminative features and avoids fixed-interval sampling.Source: §3 · p. 2, §3.1 · p. 4

3.1. Framework and Analysis

  • LAN represents pooling as feature aggregation followed by local normalization of importance weights within each sliding window.Its importance map has the same spatial size as the input and is non-negative.Source: §3.1 · p. 3
  • Average pooling assigns equal importance, while max pooling selects the largest activation; the paper argues both can miss discriminative features or hinder optimization.Average pooling can blur small discriminative features; max pooling relies on the maximum being most discriminative and sends backpropagation gradients only to local maxima.Source: §3.1 · p. 3
  • Strided convolution samples one fixed location per window, which can limit shift invariance and discard other features.Minor shifts or distortions may therefore cause large changes in downsampled features.Source: §3.1 · p. 4

3.2. Local Importance-based Pooling

  • LIP learns importance from input features and uses it to adaptively emphasize discriminative features during downsampling.A tiny fully convolutional logit module produces the importance map; exponentiating its output makes the weights non-negative.Source: §3.2 · p. 4

3.3. Exemplars: LIP-ResNet and LIP-DenseNet

  • LIP replaces selected downsampling layers in ResNet and DenseNet while leaving each architecture’s global average pooling unchanged.The ResNet design uses a 3 × 3 LIP window followed by a 1 × 1 convolution to preserve receptive fields and avoid interval sampling; DenseNet replaces its transition-block average pooling and bottom max pooling.Source: §3.3 · p. 5

4. Experiments

The experiments evaluate LIP on ImageNet classification and MS COCO detection, using ResNet and DenseNet variants and standard detection frameworks. LIP improves reported classification and detection results, with its strongest stated result on small-object detection under single-scale testing.

4.1. ImageNet Classification Experiment Setup

  • ImageNet evaluation uses single-crop validation; ResNet and DenseNet training uses mini-batches of 256, with LIP initialized to behave like average pooling.The LIP initialization sets the last convolution in each logit module to zero.Source: §4.1 · p. 5

4.2. Results on ImageNet and Analysis

  • LIP-ResNet with Bottleneck-128 improves ImageNet top-1 by 1.79% and top-5 by 0.81% over the plain network.The paper adopts Bottleneck-128 as its default logit module; instance normalization and amplified sigmoid together improve accuracy by enabling stable approximation of max pooling.Source: §4.2 · p. 6
  • Res4 is the most effective LIP placement; Combination C is recommended for practical use with fewer parameters and only 3% extra FLOPs over vanilla ResNet.More LIPs generally improve results, although placements do not contribute equally; the paper uses full-LIP Combination A for the remaining experiments.Source: §4.2 · pp. 6–7, Table 3 · p. 6
  • LIP-ResNet-101 surpasses vanilla ResNet-152 by 0.84% top-1 and 0.38% top-5, while LIP results are also favorable for DenseNet.LIP-ResNet-50 performs comparably to vanilla ResNet-101 with about half the parameters and fewer FLOPs.Source: §4.2 · p. 7, Table 4 · p. 7
  • LIP’s effective receptive field is more compact and foreground-focused than those of average pooling and strided convolution, including when gradients are traced outside the foreground.The comparison uses CAMs and backpropagated gradients in Figure 5; the latter methods show more interference from background clutter.Source: §4.2 · p. 7, Figure 5 · p. 8

4.3. MS COCO Detection Experiment Setup

  • COCO experiments train Faster R-CNN with FPN and RetinaNet on COCO 2017 train using pretrained backbones and the typical 2× schedule, with single-scale testing.The setup follows the mmdetection defaults, including an 800-pixel shorter image side, horizontal flipping, and ROI Align.Source: §4.3 · p. 7

4.4. Results on MS COCO and Analysis

  • LIP-ResNet-50 and -101 improve Faster R-CNN AP over baselines by 1.5% and 2.3%, while their APs gains over vanilla ResNets are 2.1% and 3.1%.The single-stage RetinaNet results also validate the effectiveness of LIP.Source: §4.4 · pp. 7–8, Table 5 · p. 7
  • LIP-ResNet-101-MD reaches 43.9% AP and 25.4% APs, setting a new small-object detection state of the art under single-scale testing.The comparison uses Faster R-CNN with FPN; Table 6 reports results on COCO test-dev.Source: §4.4 · p. 8, Table 6 · p. 8

5. Conclusion and Future Work

The paper presents LAN as a local-importance framework for analyzing pooling and proposes LIP to learn adaptive importance maps for downsampling. It reports benefits on ImageNet and COCO, especially for small-object detection, and identifies implementation and task coverage for future study.

Conclusion and Future Work

  • LIP-ResNet backbones improve performance for both one- and two-stage COCO detectors, especially on small objects, and reach state-of-the-art small-object detection with a standard framework.On ImageNet, LIP captures rich details for holistic image recognition.Source: §5 · p. 8
  • LIP learns adaptive, discriminative importance maps to aggregate features during downsampling, with the aim of preserving details, particularly those of tiny objects.The paper frames LAN as a local-importance framework for analyzing pooling layers and their importance maps.Source: §5 · p. 8
  • Future work will study LIP implementation, including logit-module design and adaptive pooling sizes, and test it on pose estimation and image segmentation.Source: §5 · p. 8

Related papers

Loading 1908.04156v3…