Source-linked AI summary
Improved Bilinear Pooling with CNNs
Tsung-Yu Lin, Subhransu Maji
TL;DR
The paper investigates how to normalize bilinear-pooled CNN statistics to improve their representation power. It proposes matrix square-root normalization with improved gradient and computation schemes, finding consistent accuracy gains, including 2-3% average improvement across fine-grained datasets.
Problem
The paper investigates which normalization schemes best improve the representation power and accuracy of bilinear-pooled CNN statistics.
Method
The paper adds matrix-normalization layers, evaluates matrix square-root and logarithm normalizations, and computes square roots and gradients using Lyapunov and iterative methods.
Results
Matrix square-root normalization consistently improves accuracy, outperforming matrix logarithm normalization in combination with elementwise square-root normalization and yielding 2-3% average gains across fine-grained datasets.
Takeaways & Limitations
A few Newton iterations can match SVD classification accuracy while enabling faster GPU computation, and Lyapunov gradients provide numerically stable gradient computation.
Takeaways & Limitations
SVD-based gradients are problematic for matrices with close eigenvalues, and the impact of truncated-gradient exceptions when training from scratch remains unclear.
Abstract
from arXiv · showhide
Bilinear pooling of Convolutional Neural Network (CNN) features [22, 23], and their compact variants [10], have been shown to be effective at fine-grained recognition, scene categorization, texture recognition, and visual question-answering tasks among others. The resulting representation captures second-order statistics of convolutional features in a translationally invariant manner. In this paper we investigate various ways of normalizing these statistics to improve their representation power. In particular we find that the matrix square-root normalization offers significant improvements and outperforms alternative schemes such as the matrix logarithm normalization when combined with elementwise square-root and l2 normalization. This improves the accuracy by 2-3% on a range of fine-grained recognition datasets leading to a new state of the art. We also investigate how the accuracy of matrix function computations effect network training and evaluation. In particular we compare against a technique for estimating matrix square-root gradients via solving a Lyapunov equation that is more numerically accurate than computing gradients via a Singular Value Decomposition (SVD). We find that while SVD gradients are numerically inaccurate the overall effect on the final accuracy is negligible once boundary cases are handled carefully. We present an alternative scheme for computing gradients that is faster and yet it offers improvements over the baseline model. Finally we show that the matrix square-root computed approximately using a few Newton iterations is just as accurate for the classification task but allows an order-of-magnitude faster GPU implementation compared to SVD decomposition.
1 Introduction
The paper improves B-CNN representations by normalizing bilinearly pooled CNN statistics with matrix functions. Matrix square-root normalization, especially combined with elementwise square-root normalization, improves fine-grained recognition accuracy while motivating efficient and stable computation.
- B-CNNs aggregate higher-order CNN activation statistics and have become state-of-the-art for texture and fine-grained recognition.
- Matrix-function normalization addresses the limited exploration of normalized second-order features caused by expensive and numerically unstable matrix-logarithm computation.
- 2-3% improvement in accuracy over baseline B-CNNs on three fine-grained recognition datasets comes from combining matrix square-root and elementwise square-root normalization.
- Matrix normalization scales covariance-matrix eigenvalues, complementing elementwise normalization schemes such as signed square-root normalization.
- The improved architecture adds log(A) or A1/2, signed square-root, and ℓ2 normalization layers after bilinear pooling.
2 Method and related work
The improved B-CNN applies matrix-function normalization to bilinear-pooled CNN features and studies SVD-, Lyapunov-, and Newton-based computation of these functions and their gradients.
- Bilinear pooling: Bilinear pooling globally averages locationwise outer products, producing a covariance matrix that captures pairwise feature interactions.
- Matrix-function normalization: The improved architecture applies matrix logarithm or fractional matrix-power normalization after pooling, with the matrix square-root corresponding to p = 1/2.Matrix functions depend on the entire pooled matrix, unlike elementwise transformations.
- SVD computation: SVD computes matrix functions as Z = f(A) = Ug(Σ)UT, with g applied elementwise to the diagonal singular values.The resulting gradients use derivatives with respect to the SVD factors and the diagonal function values.
- Lyapunov gradients: Lyapunov gradients derive from differentiating ZZ = A, relating changes in the square root Z to changes in A.The Lyapunov equation admits a closed-form Kronecker/vec expression and can be solved more efficiently with the Bartels-Stewart algorithm.
- Numerical stability: SVD gradients become problematic when eigenvalues are close, while truncated SVD can still produce numerical exceptions and has unclear effects during training from scratch.Lyapunov gradients are described as numerically stable and comparably efficient to SVD gradients.
- Newton iterations: Newton-based square-root iterations replace expensive matrix decompositions or inverses with matrix multiplications, enabling an order-of-magnitude faster GPU implementation under a convergence condition.The modified iterations are locally convergent when ||A − I||2 < 1 and can trade accuracy for speed by using few iterations.
3 Experiments
Experiments evaluate normalization schemes, fine-tuning, matrix-square-root computation, and comparisons with prior methods on fine-grained recognition datasets. Matrix square-root normalization consistently improves accuracy, while iterative computation preserves accuracy with substantially lower evaluation cost.
- Experimental setup: Experiments use Caltech-UCSD birds, Stanford cars, and FGVC aircrafts fine-grained recognition datasets with VGG-M and VGG-D B-CNNs.The setup follows prior B-CNN protocols, including 448×448 image resizing and dataset-specific preprocessing.
- Feature normalization: Matrix square-root normalization combined with elementwise signed square-root normalization gives the best non-fine-tuned results and generally outperforms matrix logarithm normalization.The two normalization schemes are complementary, whereas either matrix normalization alone does not always improve over the baseline.
- Feature normalization: p = 1/2, corresponding to matrix square-root normalization, works best and substantially outperforms the baseline B-CNN for non-fine-tuned VGG-D networks.The baseline p = 1 uses only elementwise signed square-root and ℓ2 normalization.
- Network fine-tuning: 2-3% average accuracy improvement is obtained after fine-tuning across the fine-grained datasets for both VGG-M and VGG-D networks.The improvement is especially large for VGG-M, using Lyapunov-computed gradients.
- Exact computations: 5 Newton iterations match SVD accuracy while being 5× faster, and one iteration improves over the baseline while taking less than 1 millisecond.Iterative matrix-square-root computation removes matrix-normalization layers as the evaluation bottleneck.
- Exact computations: SVD gradients are orders of magnitude less precise than Lyapunov gradients, yet their fine-tuning accuracy loss is negligible.Ignoring the matrix-square-root layer during fine-tuning performs worse, though it usually remains above the fine-tuned baseline B-CNN.
- Exact computations: Matrix-logarithm fine-tuning with SVD-based gradients was unsuccessful even with double precision, whereas matrix-square-root experiments used single precision.The authors suggest numerical issues are partly related to logarithmic eigenvalue scaling.
Conclusion
The paper proposes an improved B-CNN with matrix-normalization layers and evaluates square-root and logarithm normalizations, gradient computation, and iterative approximations. Matrix square-root normalization consistently improves accuracy, while a few Newton iterations provide efficient GPU computation without sacrificing classification accuracy.
- Conclusion: The improved B-CNN adds matrix-normalization layers that provide complementary benefits to elementwise normalization and consistently improve over the baseline.Matrix square-root normalization generally outperforms matrix logarithm normalization when combined with elementwise square-root normalization.
- Conclusion: A few Newton iterations compute the matrix square-root efficiently on a GPU, while Lyapunov equations provide accurate gradients.Even a single iteration improves over the baseline with negligible added forward-computation cost.