PaperlayerView exact revision on arXiv

Source-linked AI summary

Is Second-order Information Helpful for Large-scale Visual Recognition?

Peihua Li, Jiangtao Xie, Qilong Wang, Wangmeng Zuo

arXiv:1703.08050v3cs.CV

TL;DR

The paper asks whether visual recognition can benefit from feature statistics beyond first order, given the difficulty of estimating high-dimensional covariances and using their manifold structure. It proposes end-to-end Matrix Power Normalized Covariance pooling and reports competitive ImageNet gains over first-order counterparts across multiple ConvNet architectures.

  • Problem

    Large-scale ConvNets rarely exploit feature statistics beyond first order, although modeling high-dimensional feature distributions is important for recognition.

  • Method

    MPN-COV pools high-level convolutional features into covariance matrices, uses matrix power normalization, and supplies forward and backward formulas for end-to-end training.

  • Results

    MPN-COV networks achieved competitive ImageNet 2012 gains over first-order counterparts across AlexNet, VGG, and ResNet architectures.

  • Takeaways & Limitations

    Second-order covariance representations can improve large-scale visual recognition relative to first-order pooling under the evaluated ConvNet architectures.

  • Takeaways & Limitations

    The Log-Euclidean comparison depends on selecting a small positive eigenvalue offset ϵ, whose suitable value is difficult to determine across many images.

Abstract

from arXiv · show

By stacking layers of convolution and nonlinearity, convolutional networks (ConvNets) effectively learn from low-level to high-level features and discriminative representations. Since the end goal of large-scale recognition is to delineate complex boundaries of thousands of classes, adequate exploration of feature distributions is important for realizing full potentials of ConvNets. However, state-of-the-art works concentrate only on deeper or wider architecture design, while rarely exploring feature statistics higher than first-order. We take a step towards addressing this problem. Our method consists in covariance pooling, instead of the most commonly used first-order pooling, of high-level convolutional features. The main challenges involved are robust covariance estimation given a small sample of large-dimensional features and usage of the manifold structure of covariance matrices. To address these challenges, we present a Matrix Power Normalized Covariance (MPN-COV) method. We develop forward and backward propagation formulas regarding the nonlinear matrix functions such that MPN-COV can be trained end-to-end. In addition, we analyze both qualitatively and quantitatively its advantage over the well-known Log-Euclidean metric. On the ImageNet 2012 validation set, by combining MPN-COV we achieve over 4%, 3% and 2.5% gains for AlexNet, VGG-M and VGG-16, respectively; integration of MPN-COV into 50-layer ResNet outperforms ResNet-101 and is comparable to ResNet-152. The source code will be available on the project page: http://www.peihuali.org/MPN-COV

1. Introduction

The paper argues that large-scale visual recognition should exploit feature-distribution statistics beyond first order, and proposes MPN-COV to make covariance pooling robust and trainable end-to-end.

  • ConvNets learn high-level features for complex decision boundaries, making adequate consideration of their feature distributions important.
  • Most ConvNet research emphasizes deeper or wider architectures while rarely exploiting statistical information beyond first order.
  • MPN-COV replaces common first-order pooling with covariance pooling of high-level convolutional features.
  • Covariance pooling must address robust estimation from few high-dimensional features and the manifold structure of covariance matrices.
  • MPN-COV develops nonlinear matrix-function gradients and forward formulas, enabling end-to-end training.
  • The paper interprets MPN-COV statistically, geometrically, and computationally, and evaluates it across ImageNet ConvNet architectures.

2. Related Work

Prior work established higher-order feature statistics and end-to-end structured pooling, including covariance- and outer-product-based approaches for visual recognition.

  • Higher-order statistics had already improved classical and deep-learning classification using descriptors, Fisher Vectors, and Fisher Information matrices.
  • O2P introduced second-order pooling for image classification but used non-central second-order moments subject to a matrix logarithm.
  • Higher-order pooling was also explored through second- and third-order descriptors, power-normalized autocorrelation structures, and tensor descriptors for action recognition.
  • DeepO2P enabled end-to-end second-order pooling through SVD, while B-CNN aggregated outer products from two convolutional networks with element-wise power normalization.
  • MPN-COV is presented within this line of work as another structured approach to second-order convolutional-feature representations.

3. The Proposed MPN-COV

MPN-COV converts last-layer convolutional responses into a normalized covariance representation, applies matrix power through eigendecomposition, and propagates gradients through the resulting nonlinear operations.

  • 3. The Proposed MPN-COV: MPN-COV represents an image with a normalized covariance matrix characterizing feature-channel correlations and feature-distribution shape.
  • 3.1. Forward Propagation: The forward pipeline computes a sample covariance matrix, eigendecomposes it, applies a power to its eigenvalues, and sends the resulting matrix to the FC layer.
  • 3.1. Forward Propagation: The covariance matrix is formed from a sample X of N d-dimensional features and is symmetric positive semi-definite.
  • 3.1. Forward Propagation: Eigendecomposition represents the covariance matrix with ordered eigenvalues and corresponding orthogonal eigenvectors, enabling matrix powers to be computed spectrally.
  • 3.1. Forward Propagation: Optional matrix ℓ2 or Frobenius normalization can follow matrix power normalization, with α = 1 recovering the separate norm normalizations.
  • 3.2. Backward Propagation: Matrix backpropagation handles nonlinear matrix functions while preserving orthogonality, diagonality, and symmetry constraints.
  • 3.2. Backward Propagation: The backward pass applies the chain rule from the FC output through eigendecomposition and ultimately derives gradients with respect to the input features.

4. The Mechanism Underlying MPN-COV

MPN-COV addresses small-sample, high-dimensional covariance estimation and covariance-manifold geometry through eigenvalue power normalization. Its analysis contrasts power normalization with logarithmic normalization, showing that the former preserves eigenvalue significance while the latter can reverse it.

  • 4.1. MPN-COV Amounts to Robust Covariance Estimation: Small-sample covariance estimation is unreliable when feature dimension is large relative to the number of convolutional features.The paper notes that last convolutional layers can output fewer than 200 samples with feature dimension greater than 512.
  • 4.1. MPN-COV Amounts to Robust Covariance Estimation: MPN-COV applies individualized eigenvalue shrinkage, reducing large sample eigenvalues and stretching small ones according to the covariance spectrum.This behavior closely follows the shrinkage principle used for robust covariance estimation.
  • 4.1. MPN-COV Amounts to Robust Covariance Estimation: MPN-COV acts as a robust covariance estimator derived from regularized maximum likelihood with a penalty encouraging similarity to the identity matrix.The regularization adds a matrix von Neumann divergence term beyond the classical maximum-likelihood objective.
  • 4.2. MPN-COV Approximately Exploits Riemannian Geometry: The covariance-matrix space is a Riemannian manifold, but affine-invariant metrics are computationally inefficient, motivating the scalable Log-Euclidean and Power-Euclidean alternatives.The Power-Euclidean metric approaches the Log-Euclidean metric as the power exponent approaches zero.
  • 4.3. Qualitative Analysis: AlexNet Conv5 covariances have fewer than 169 nonzero eigenvalues because each covariance is estimated from 169 spatial features.The experiment uses 300,000 ImageNet training images and computes covariance eigenvalues from 13 × 13 Conv5 feature maps with 256 channels.
  • 4.3. Qualitative Analysis: Logarithmic normalization can reverse eigenvalue significance by over-stretching small eigenvalues, whereas power normalization preserves their original significance ordering.For example, log normalization maps λ = 50 to approximately 3.9 and λ = 10^-3 to approximately -6.9; its derivative is also much larger for the smaller eigenvalue.

5. Experiments

Experiments on ImageNet evaluate MPN-COV across AlexNet and VGG architectures, normalization choices, initialization strategies, and existing second-order pooling methods. MPN-COV consistently improves recognition performance over first-order pooling and competing methods.

  • Experimental setup: On ImageNet 2012, experiments report top-1 and top-5 validation error rates using no extra training images.The dataset contains 1,000 classes, roughly 1.28 million training images, 50,000 validation images, and 100,000 test images.
  • AlexNet evaluation: 40.41% top-1 error (1-crop) is achieved by plain covariance pooling under AlexNet, about 1.1% below first-order max pooling.For α < 1, decreasing α initially reduces error, but performance later worsens; results vary insignificantly over α ∈ [0.4, 0.9].
  • AlexNet evaluation: MPN improves AlexNet top-1 error by ∼1.3% over M-Fro and ∼1.1% over M-ℓ2, while combining these normalizations with MPN degrades performance.Element-wise power normalization after MPN also degrades performance by over 1%.
  • AlexNet evaluation: ∼1.1% lower error is obtained with warm initialization than with random initialization for the MPN-COV network.The authors associate this reduction with convergence to a better local minimum of the loss function.
  • Comparison with second-order methods: MPN-COV outperforms B-CNN by 1.4% and is also better than DeepO2P under AlexNet with 1-crop evaluation.B-CNN improves plain covariance pooling by ∼0.5%, whereas DeepO2P is reported as inferior to plain covariance pooling.
  • VGG-Net architectures: Under VGG-M, MPN-COV gains ∼2.5% with random initialization and ∼3.6% with warm initialization over first-order max pooling in configuration a.Configuration b yields gains of ∼2% and ∼3% under random and warm initialization, respectively; MPN-COV also brings large improvement under VGG-16.
  • VGG-Net architectures: ∼2.7% lower top-1 error is reported for MPN-COV VGG-16 versus original VGG-16 in 10-crop comparison.Its top-5 error is also better than GoogleNet by ∼1.4% and PReLU-net B by ∼0.4%.

6. Conclusion

The paper proposes MPN-COV to exploit second-order statistics in large-scale classification, addressing covariance estimation and covariance-matrix geometry. Experiments on ImageNet 2012 report competitive gains over first-order counterparts.

  • MPN-COV explores second-order statistics through matrix-normalized covariance representations for large-scale classification.It targets robust covariance estimation from few high-dimensional features while approximately exploiting covariance-space geometry.
Loading 1703.08050v3…