PaperlayerView exact revision on arXiv

Source-linked AI summary

Global Second-order Pooling Convolutional Networks

Zilin Gao, Jiangtao Xie, Qilong Wang, Peihua Li

arXiv:1811.12006v2cs.CV

TL;DR

ConvNets need stronger higher-order representations, but how to use them effectively in earlier layers remains open. The paper introduces GSoP blocks across network stages for holistic channel- and position-wise modeling, and reports improvements over counterparts and state-of-the-art results.

  • Problem

    Effectively introducing higher-order representations into earlier ConvNet layers remains an open problem despite successful network-end GSoP.

  • Method

    The method inserts modular GSoP blocks across ConvNet stages to transform holistic channel or spatial covariance statistics into tensor-scaling weights.

  • Results

    GSoP-Net outperformed counterparts, including SE-Net and iSQRT-COV, with reported gains on CIFAR-100 and comparisons against competing networks.

  • Takeaways & Limitations

    Introducing holistic second-order pooling at intermediate stages can improve discriminative representations beyond using global covariance pooling only at the network end.

Abstract

from arXiv · show

Deep Convolutional Networks (ConvNets) are fundamental to, besides large-scale visual recognition, a lot of vision tasks. As the primary goal of the ConvNets is to characterize complex boundaries of thousands of classes in a high-dimensional space, it is critical to learn higher-order representations for enhancing non-linear modeling capability. Recently, Global Second-order Pooling (GSoP), plugged at the end of networks, has attracted increasing attentions, achieving much better performance than classical, first-order networks in a variety of vision tasks. However, how to effectively introduce higher-order representation in earlier layers for improving non-linear capability of ConvNets is still an open problem. In this paper, we propose a novel network model introducing GSoP across from lower to higher layers for exploiting holistic image information throughout a network. Given an input 3D tensor outputted by some previous convolutional layer, we perform GSoP to obtain a covariance matrix which, after nonlinear transformation, is used for tensor scaling along channel dimension. Similarly, we can perform GSoP along spatial dimension for tensor scaling as well. In this way, we can make full use of the second-order statistics of the holistic image throughout a network. The proposed networks are thoroughly evaluated on large-scale ImageNet-1K, and experiments have shown that they outperformed non-trivially the counterparts while achieving state-of-the-art results.

1. Introduction

ConvNets benefit from higher-order representations, but effectively introducing them before the network end remains an open problem. The paper proposes GSoP blocks throughout deep ConvNets to exploit holistic second-order statistics earlier.

  • Higher-order representations can enhance ConvNets’ nonlinear modeling capability for characterizing complex class boundaries.
  • Global second-order pooling produces covariance-based image representations and has achieved state-of-the-art results across several vision tasks.
  • Introducing higher-order representations in earlier layers remains an open problem, despite successful network-end GSoP.
  • The proposed GSoP blocks are inserted from lower to higher layers to exploit holistic image second-order statistics throughout the network.
  • Given a previous-layer 3D tensor, the method models pairwise channel correlations, embeds the covariance matrix, and uses it to scale channels.
  • The modular blocks can capture long-range statistical dependencies while maintaining low memory and computational complexity.

2. Related Works

Prior work mainly applies first- or second-order global pooling at the network end, or models higher-order interactions locally. GSoP-Net instead introduces holistic second-order pooling into intermediate ConvNet layers.

  • Global average pooling summarizes first-order mean statistics and is widely used at network ends, while SE-Net brings it into intermediate layers.
  • Network-end GSoP uses trainable global second-order pooling and covariance representations for improved image modeling.
  • GSoP-Net introduces global second-order pooling into intermediate layers to capture holistic image information throughout a network.
  • Existing quadratic transformation networks model feature interactions through nonlinear filters applied to local neighborhoods.
  • The GSoP block computes a covariance matrix from an input tensor, transforms it, and produces tensor scaling along the channel dimension.

3. Global Second-order Pooling Network

GSoP-Net inserts channel-wise or position-wise second-order pooling into ConvNets to model holistic dependencies and generate nonlinear scaling weights. Two architectures and fixed-size implementations balance representation quality with computational cost.

  • Global Second-order Pooling Network: GSoP blocks can be inserted after convolutional layers to model holistic high-order statistics at early network stages.
  • Global Second-order Pooling Network: GSoP-Net1 uses intermediate and final GSoP blocks followed by global average pooling, whereas GSoP-Net2 uses matrix power-normalized covariance representations at the network end.
  • 3.1. Global Second-order Pooling Block: The channel-wise block reduces channels, computes a c × c covariance matrix, row-normalizes it, and interprets each row as one channel’s dependencies.
  • 3.1. Global Second-order Pooling Block: Two convolution-plus-activation operations embed covariance statistics and produce soft channel weights that emphasize or suppress individual channels.
  • 3.2. Extension to Spatial Position: Position-wise GSoP models long-range dependencies that ordinary convolutions miss because limited receptive fields restrict distant feature interactions.
  • 3.2. Extension to Spatial Position: Position-wise GSoP computes an hw × hw covariance matrix, transforms it into a spatial weight matrix, upsamples it, and scales features across positions.
  • 3.4. Block Implementation: Implementations use fixed covariance sizes, reducing channels to 128 and spatial maps to 8 × 8 to trade computational complexity against accuracy.

4. Experiments

Experiments on ImageNet-1K and CIFAR-100 evaluate GSoP-Nets through ablations, convergence, complexity, and comparisons with competing architectures. Results consistently show benefits from introducing second-order statistics in earlier network stages, with gains accompanied by specific parameter and computation costs.

  • Experimental setup: Experiments use ImageNet-1K, a 250-class ImageNet-1K subset for ablations, and CIFAR-100 to evaluate generalization.The standard ImageNet-1K benchmark contains 1.28M training images and 50K validation images; CIFAR-100 contains 50K training and 10K testing images.
  • Ablation analysis: GSoP-Net variants improve over vanilla ResNet across channel-wise and position-wise pooling settings, with GSoP-Net2 consistently outperforming GSoP-Net1.Best reported covariance sizes are c = 128 for GSoP-Net1 and c = 256 for GSoP-Net2 in channel-wise pooling; GSoP-Net1 performs best with hw = 64 for position-wise pooling.
  • Ablation analysis: Combining position-wise with channel-wise pooling brings little improvement, indicating that the two second-order pooling methods are not complementary in these experiments.For GSoP-Net1, maximum fusion is slightly better than concatenation; for GSoP-Net2, concatenation is slightly superior, but fusion adds little over separate channel-wise pooling.
  • Ablation analysis: Single channel-wise GSoP blocks inserted at different residual stages provide comparable improvements, while GSoP-Net2 outperforms the strong iSQRT-COV baseline.The ablation supports introducing second-order statistics at intermediate layers rather than using them only for the final image representation.
  • Convergence and complexity: GSoP-Net1 converges similarly to ResNet-50 but maintains lower validation error, while GSoP-Net2 inherits iSQRT-COV’s fast convergence and performs steadily better.The authors attribute these improvements to holistic modeling of second-order statistics in earlier stages.
  • Convergence and complexity: GSoP-Net1 has parameter count comparable to vanilla ResNet-50, whereas GSoP-Net2 nearly doubles parameters and both networks require 1.58x vanilla ResNet GFLOPs.GSoP-Net2’s parameter increase mainly comes from its 32K-dimensional image representation and fully connected layer; removing final-stage downsampling increases computation.
  • Competing networks: GSoP-Net2 outperforms MPN-COV by 1.55% in top-1 error and 0.90% in top-5 error, and achieves 0.95%/0.58% lower top-1/top-5 error rates than iSQRT-COV.The improvement over iSQRT-COV is reported with negligible overhead.
  • Competing networks: GSoP-Net1 achieves 1.0%/0.6% better top-1/top-5 errors than SE-Net, while GSoP-Net2 improves over iSQRT-COV by 1.37% on CIFAR-100.The comparisons also report lower error rates than CBAM and a non-trivial margin over GENet.

5. Conclusion

The paper presents a deep convolutional model that captures holistic statistical correlations across all network stages. Its GSoP blocks improve representations, outperform SE-Net and iSQRT-COV, and can be plugged into other architectures.

  • The proposed model captures holistic statistical correlations across all stages of the network.
  • Introducing higher-order information at earlier stages helps the model learn more discriminative representations.
  • The proposed networks outperform SE-Net, the first-order counterpart, and non-trivially improve iSQRT-COV, which uses global covariance pooling only at network end.
  • GSoP blocks are modular and can be plugged into architectures such as Inception and DenseNet.
Loading 1811.12006v2…