Deep CNNs Meet Global Covariance Pooling: Better Representation and Generalization

• Global average pooling captures only first-order statistics, limiting the representation and generalization abilities attributed to deep CNNs.Source: §1 · p. 1
• Earlier deep global covariance-pooling methods leave robust estimation or covariance geometry unresolved, with LERM also reported to have a side effect in the high-dimensional, small-sample setting.Source: §1 · p. 2
• MPN-COV combines robust covariance estimation for high-dimensional, small-sample features with matrix-power normalization that exploits covariance geometry.The introduction relates these properties to vN-MLE and the Power-Euclidean metric.Source: §1 · p. 2
• The proposed refinements add first-order information through Gaussian embedding, use Newton-Schulz iteration for faster square-root normalization, and compress covariance representations with progressive 1 × 1 and group convolutions.The introduction says pre-normalization and post-compensation are needed for convergence of deeper CNNs such as ResNet.Source: §1 · pp. 2–3
• Experiments across object, scene, fine-grained, and texture recognition show the proposed GCP networks outperform their counterparts.The evaluations use multiple CNN architectures, including AlexNet, VGG-VD, ResNet, and DenseNet.Source: §1 · p. 3

2 RELATED WORK

• The section groups work on trainable sophisticated pooling or encoding methods into four categories.Source: §2 · p. 3

2.1 Deep CNNs with Global Second-order Pooling

• DeepO2P applies matrix-log normalization and B-CNN applies element-wise power followed by ℓ2 normalization; the paper positions MPN-COV as adding theoretical explanations and large-scale ImageNet experiments for matrix power normalization.The related work also distinguishes iSQRT-COV by its pre-normalization, coupled iteration, and post-compensation, and says its forward and backward passes use Newton-Schulz iteration.Source: §2.1 · p. 3

2.2 Deep CNNs with Global Approximate High-order Pooling

• Compact B-CNN, low-rank B-CNN, and related high-order methods approximate representations to reduce their size, while the cited methods do not address robust estimation or manifold geometry.SMSO is described as producing compact second-order representations with state-of-the-art results.Source: §2.2 · p. 3

2.3 Deep CNNs with Local Second-order Statistics

• Factorized Bilinear adds a parametric quadratic term, while SORT fuses two-branch outputs with a second-order term; both lines of work are described as discarding robust estimation and geometry.Source: §2.3 · pp. 3–4

2.4 Deep CNNs with Trainable BoVW Methods

• Trainable BoVW approaches include NetVLAD, FisherNet, and MFAFVNet, whereas this paper proposes global matrix power normalized covariance pooling.Source: §2.4 · p. 4

3.1.1 Computation of MPN-COV

• MPN-COV applies matrix power to sample covariance as a global second-order representation of the last convolution’s spatial features.The feature tensor is reshaped into d-dimensional samples before covariance computation; the power operation uses eigenvalues raised to α.Source: §3.1.1 · p. 4

3.1.2 Robust Covariance Estimation

• In high-dimensional, small-sample settings, sample covariance estimates are ill-conditioned; MPN-COV shrinks large eigenvalues and stretches small ones, with α = 1/2 providing robust estimation.The paper reports that α = 1/2 performs best in Section 4.2.1.Source: §3.1.2 · pp. 4–5

3.1.3 Approximate Usage of Geometry

• MPN-COV can be viewed as using the Power-Euclidean metric to approximately exploit covariance geometry, and this metric approaches the LERM as α approaches zero.The passage characterizes LERM as efficient and scalable, while noting that Power-Euclidean measures geodesic distance approximately.Source: §3.1.3 · p. 5

3.2 Global MPN-COV Pooling Network

• MPN-COV-Net replaces global average pooling after the last convolution with covariance pooling followed by matrix power normalization to collect global second-order feature statistics.The block can additionally apply matrix ℓ2-norm or Frobenius-norm normalization.Source: §3.2 · p. 5

3.3 Global Gaussian Embedding Network

• The Gaussian embedding network combines first- and second-order feature statistics by mapping a Gaussian to the square root of an SPD matrix, enabling backpropagation.The embedding equips Gaussian space with a Lie group structure and is implemented as matrix-partition and square-root SPD-matrix layers.Source: §3.3 · pp. 5–6

3.4 Iterative Matrix Square Root Normalized Covariance Pooling Network

• iSQRT-COV uses Newton-Schulz matrix iterations with pre-normalization and post-compensation; post-compensation yields about 1% lower top-1 error than batch normalization on ResNet-50 ImageNet.Pre-normalization addresses the iteration’s local convergence condition, while post-compensation counters adverse effects on network convergence.Source: §3.4 · pp. 6–7

3.5 Compact Covariance Representations

• To reduce covariance-representation size, the compact strategy uses progressive 1 × 1 convolutions before covariance computation and group convolution after the global covariance pooling block.For d = 512 and d = 1024, the vectorized covariance representations contain 131,328 and 524,800 elements, respectively.Source: §3.5 · pp. 7–8
• Progressive 1 × 1 convolutions reduce input dimensions from d to ˆd, avoiding abrupt reduction and outperforming a single 1 × 1 convolution.The resulting covariance representation has size ˆd × (ˆd + 1)/2; for ˆd = 128, that is 8256.Source: §3.5 · p. 8
• Group convolution divides covariance representation z into G groups, reducing the parameters of its learned linear transform from dz × dˆz to (dz × dˆz)/G.The transform corresponds to a fully connected layer after covariance pooling, and grouping is proposed to reduce its computation and memory costs.Source: §3.5 · p. 8

4.1 Implementation Details

• The evaluation spans large- and small-scale recognition datasets, with ImageNet ablations examining normalization, iteration count, training speed, and compact representations.The implementation inserts GCP after the last convolution layer; most architectures use a 1×1 convolution to reduce features to 256 channels before covariance pooling.Source: §4 · p. 8, §4.2.1 · pp. 8, 10, §4.1 · p. 8

4.2.1 Ablation Study

• Across the compared normalization methods, MPN achieves the best result, outperforming M-Fro, M-ℓ2, and E-PN by approximately 1.3%, 1.1%, and 1.3%, respectively; further normalization after MPN-COV produces negative gains.At three iterations, iSQRT-COV has 0.3% higher error than MPN-COV; it is marginally better at five to seven iterations, and the authors set N = 5. Post-compensation is essential for state-of-the-art results with deep CNNs; combining progressive 1 × 1 convolutions with group convolution yields 22.4% top-1 error with 2K-dimensional representations, outperforming original ResNet-50 with first-order GAP.Source: §4.2.1 · pp. 9–10

4.2.2 Comparisons with Different Methods

• MPN-COV improves classification over corresponding CNN baselines across AlexNet, VGG, and ResNet architectures, while iSQRT-COV also compares favorably with several competing methods.The comparisons include local second-order networks and other deep global covariance-pooling methods; on fine-grained benchmarks, pre-trained iSQRT-COV-Net with ResNet-101 establishes state-of-the-art results.Source: §4.2.2 · pp. 10–12, §4.4.1 · p. 13

4.3 Large-scale Scene Categorization on Places365

• On Places365, iSQRT-COV-Net with ResNet-50 beats original ResNet-50 by about 1.2% in Top-1 error and 1% in Top-5 error.It also outperforms B-CNN under the same settings.Source: §4.3 · p. 12, Table 7 · p. 12

4.4.1 Fine-grained Visual Recognition

• On fine-grained benchmarks, ResNet-50 iSQRT-COV-Net (8K) outperforms KP by about 2.6%, 3.8%, and 0.6% on Birds, Aircrafts, and Cars, respectively.The 32K variant further improves accuracy, and ResNet-101 iSQRT-COV-Net establishes state-of-the-art results on all three benchmarks.Source: Table 8 · p. 12, §4.4.1 · p. 13

4.4.2 Texture Classification

• On DTD and Indoor67, iSQRT-COV-Net with VGG-VD16 outperforms the original model and several deep second-order and BoVW methods.The authors report these transfer results as evidence that their GCP methods improve deep CNN generalization.Source: §4.4.2 · p. 13, Table 9 · p. 13

4.4.3 Effect of Different Fine-tuning Strategies.

• On small-scale benchmarks, iSQRT-COV-Net exceeds iSQRT-COV (PreTr-GAP), while replacing GAP with iSQRT-COV during fine-tuning also improves GAP-based pre-trained networks.The reported comparisons also show iSQRT-COV outperforming B-CNN under corresponding pre-training settings; E-PN has little effect when fine-tuning iSQRT-COV-Net.Source: §4.4.3 · p. 13, Table 10 · p. 13

4.5 iNaturalist Challenge 2018

• On iNaturalist, fusing iSQRT-COV-Net and ResNet-152 models places first and beats the runner-up by about 1.2%.An ensemble of iSQRT-COV-Net models alone also beats the runner-up by about 0.8%.Source: §4.5 · p. 14, Table 11 · p. 14

5 CONCLUSION

• MPN-COV is proposed to improve deep CNN representation and generalization by robustly estimating covariance and using covariance geometry.The method is presented as a global pooling methodology with statistical and geometrical foundations.Source: §5 · p. 14
• Gaussian embedding adds first-order statistics, iterative matrix square-root normalization speeds training, and compact covariance strategies reduce model complexity.The conclusion reports that the resulting methods improve global covariance pooling while retaining fast training and affordable model complexity.Source: §5 · p. 14
• The authors identify object detection and semantic segmentation as future applications for the proposed methods.Source: §5 · p. 14

TION OF ISQRT-COV BLOCK

• The appendix derives gradients for structured iSQRT-COV layers using matrix backpropagation so the block can be implemented without autograd.The authors note that toolkit gradients can be a black box and that PyTorch 0.3.0 or below may compute them incorrectly.Source: TION OF ISQRT-COV BLOCK · pp. 14–15
• The Newton-Schulz backward gradients are computed through the iteration's symmetric matrices and matrix-backpropagation chain rules.Source: TION OF ISQRT-COV BLOCK · p. 14
• For pre-normalization, the appendix combines the loss gradient with respect to Σ from the post-compensation layer and derives gradients for the normalization choice.It gives trace normalization as an example and also addresses Frobenius-norm pre-normalization.Source: TION OF ISQRT-COV BLOCK · p. 15

APPENDIX II: COMPARISON OF DIFFERENT GAUSSIAN EMBEDDING METHODS FOR G2DENET

• The paper contrasts its embedding with alternatives involving affine coordinates, Cholesky decomposition, or a unique SPD-matrix representation.The text states that the Cholesky-based embedding is unsuitable for backpropagation because it involves covariance-matrix decomposition.Source: APPENDIX II: COMPARISON OF DIFFERENT… · p. 15
• The proposed Gaussian embedding achieves the best Birds-CUB200-2011 result, outperforming competing methods by 3%–3.6%.The comparison uses VGG-VD16 as the backbone.Source: APPENDIX II: COMPARISON OF DIFFERENT… · p. 16
• The method considers both geometric and algebraic Lie-group structure when embedding Gaussian distributions.Source: APPENDIX II: COMPARISON OF DIFFERENT… · pp. 15–16

NESS USING RESNETS

• iSQRT-COV (8k) reduces Top-1 errors by about 5.3%, 3.5%, 2.4%, and 2.3% versus GAP on ResNet-18, ResNet34, ResNet-50, and ResNet-101, respectively.The comparison uses models trained under the same experimental settings with 224×224 input images.Source: NESS USING RESNETS · p. 16
• With similar or lower complexity, ResNet18, ResNet50, and ResNet101 with iSQRT-COV (8k) lower Top-1 error by 1.8%, 1.3%, and 1.7% versus ResNet34, ResNet101, and ResNet152 with GAP, respectively.The passage compares models with similar space and time complexity.Source: NESS USING RESNETS · p. 16
• At comparable performance, ResNet34 and ResNet50 with iSQRT-COV (8k) are comparable or slightly superior to ResNet101 and ResNet152 with GAP, respectively, at much lower complexity.Source: NESS USING RESNETS · p. 16
• iSQRT-COV (8k) adds about 7M parameters, 0.2 ms inference time, and 100M memory cost while reducing Top-1 error across the four ResNet backbones.Source: NESS USING RESNETS · p. 16

APPENDIX IV: COMPARISON BETWEEN SVM AND SOFTMAX CLASSIFIERS ON FINE-GRAINED VISUAL CLASSIFICATION TASK

• The comparison follows the same experimental settings as prior works and reports classification accuracy for softmax and SVM on fine-grained visual classification.Source: APPENDIX IV: COMPARISON BETWEEN SVM… · p. 16, Table 14 · p. 16
• SVM and softmax classifiers achieve very similar results for iSQRT-COV-Net, except on Aircrafts with ResNet-50, where softmax is a little better.Source: APPENDIX IV: COMPARISON BETWEEN SVM… · p. 16
• The experiments suggest that training an SVM after fine-tuning with softmax brings negligible overall benefit for iSQRT-COV-Net.Source: APPENDIX IV: COMPARISON BETWEEN SVM… · p. 16

AND ACCURACY

• Caching intermediate tensors adds iSQRT-COV memory overhead, but that overhead is less than 1% of the whole networks.The cached tensors are needed for backpropagation.Source: AND ACCURACY · p. 16
• The comparison evaluates memory, training speed, and classification accuracy on Birds-CUB200-2011 using VGG-VD16 and batch size 24.Source: Table 15 · p. 17
• iSQRT-COV training is 1.8x faster than Improved B-CNN using Newton-Schulz forward propagation and SVD+Lyapunov backward propagation.Source: AND ACCURACY · p. 17
• At five iterations, Improved B-CNN using Newton-Schulz and Lyapunov iterations has comparable training speed, but iSQRT-COV has higher classification accuracy.More iterations can improve Improved B-CNN results but slow training.Source: AND ACCURACY · p. 17