Low-rank Bilinear Pooling for Fine-Grained Classification

Shu Kong; Charless Fowlkes

Source-linked AI summary

Low-rank Bilinear Pooling for Fine-Grained Classification

Shu Kong, Charless Fowlkes

arXiv:1611.05109v2Nov 16, 2016cs.CV

TL;DR

Fine-grained bilinear models achieve strong performance but incur high-dimensional computational and parameter costs. This paper uses matrix-structured low-rank bilinear classifiers and classifier co-decomposition to compress them, achieving state-of-the-art results with substantially smaller models and category labels alone.

Problem
High-dimensional bilinear features impose substantial computational burdens and require large quantities of training data to fit.
Method
The paper represents covariance features as matrices, applies low-rank symmetric bilinear classifiers, and further factorizes class classifiers through co-decomposition.
Results
The final model achieves state-of-the-art performance on several public fine-grained datasets using only category labels and is ten times smaller than compact bilinear models and one hundred times smaller than full bilinear CNNs.
Takeaways & Limitations
Frobenius-norm scoring avoids explicit high-dimensional bilinear-feature computation, while co-decomposition further reduces computation and parameter storage.
Takeaways & Limitations
The hinge loss is convex in the full classifier but not in the optimized factor parameters, and nuclear-norm regularization would not provide the highlighted training-time computational benefits.

Abstract

from arXiv · show

Pooling second-order local feature statistics to form a high-dimensional bilinear feature has been shown to achieve state-of-the-art performance on a variety of fine-grained classification tasks. To address the computational demands of high feature dimensionality, we propose to represent the covariance features as a matrix and apply a low-rank bilinear classifier. The resulting classifier can be evaluated without explicitly computing the bilinear feature map which allows for a large reduction in the compute time as well as decreasing the effective number of parameters to be learned. To further compress the model, we propose classifier co-decomposition that factorizes the collection of bilinear classifiers into a common factor and compact per-class terms. The co-decomposition idea can be deployed through two convolutional layers and trained in an end-to-end architecture. We suggest a simple yet effective initialization that avoids explicitly first training and factorizing the larger bilinear classifiers. Through extensive experiments, we show that our model achieves state-of-the-art performance on several public datasets for fine-grained classification trained with only category labels. Importantly, our final model is an order of magnitude smaller than the recently proposed compact bilinear model, and three orders smaller than the standard bilinear CNN model.

1. Introduction and Related Work

Fine-grained recognition requires subtle distinctions under substantial visual variability, while bilinear pooling improves performance by using second-order statistics but creates high-dimensional, costly representations. The paper proposes low-rank bilinear classification and classifier co-decomposition to reduce computation and model size while retaining strong classification performance.

Motivation: Fine-grained categorization distinguishes subordinate categories despite low inter-class variance and high intra-class variance from factors such as deformable pose.The task is further challenged by many categories and limited training data.
Prior approaches: Bilinear pooling forms holistic image representations from second-order local-feature statistics, combining deformation invariance from spatial pooling with feature selectivity.It has achieved state-of-the-art performance on several fine-grained classification problems.
Prior approaches: High-dimensional bilinear features impose substantial computational and training-data burdens, motivating compact approximations such as random Maclaurin and tensor sketch models.The compact models approximate classifiers on bilinear-pooled features using projected local features and random matrices.
Low-rank bilinear pooling: The proposed method represents bilinear features as covariance matrices and applies low-rank bilinear classifiers, preserving algebraic structure while reducing parameter degrees of freedom.Unlike prior approaches that vectorize features and apply linear classifiers, this formulation exploits matrix structure.
Low-rank bilinear pooling: A symmetric bilinear form evaluates classification confidence through the squared Frobenius norm of projected local features, avoiding explicit bilinear-feature computation at test time.The paper reports improved classification performance, smaller models, and faster feedforward computation under some conditions.
Model compression: Classifier co-decomposition factorizes the joint classifier parameters into a compact shared representation and can be implemented with two separate linear convolutional layers.A feature-map-statistics initialization supports direct end-to-end training without first training and factorizing larger classifiers.
Results: State-of-the-art performance is achieved on several public fine-grained datasets using only category labels, with ten times fewer parameters than compact bilinear models and one hundred times fewer than full bilinear CNNs.The comparison concerns the final model's learned parameter set.

2. Bilinear Features Meet Bilinear SVMs

Bilinear pooling represents CNN local-feature second-order statistics as a covariance-like matrix, while bilinear SVMs exploit this structure to support symmetric, low-rank classifiers. Empirically, most classifier eigenvalues are near zero, so rank-10 approximations retain nearly full-rank accuracy with far fewer parameters and avoid explicitly computing pooled bilinear features.

Bilinear Features: CNN feature maps are pooled through local outer products, producing a covariance-like matrix of second-order activation statistics.The matrix representation has dimension c^2 before vectorization.
Bilinear SVMs: The standard bilinear CNN vectorizes XX^T and applies a linear classifier, whereas the proposed approach preserves its matrix structure.The matrix formulation enables low-rank constraints on classifier parameters.
Bilinear SVMs: The optimal classifier is symmetric because it corresponds to a sum of symmetric bilinear-feature matrices.Its dual expansion represents the classifier using training examples and their SVM dual variables.
Low-Rank Classifiers: Most classifier eigenvalues are close to zero, with magnitudes much smaller than the largest eigenvalues.This spectrum motivates low-rank parameterization of the bilinear classifier.
Low-Rank Classifiers: Rank-10 approximations achieve nearly the same classification accuracy as full-rank classifiers, using 512 × 10 × 200 rather than 512^2 × 200 parameters.The approximations are generated by singular value decomposition of the full-rank classifiers.
Optimization: The low-rank classifier evaluates a difference of matrix norms, eliminating the need to explicitly compute the pooled bilinear feature.The resulting objective is non-convex in its low-rank factors and is optimized with stochastic gradient descent for end-to-end CNN training.

3. Classifier Co-Decomposition for Model Compression

Classifier co-decomposition compresses multiple bilinear classifiers by isolating shared structure and using compact class-specific terms. The factorization can be implemented end-to-end with a projection layer and lower-dimensional classifiers, with PCA providing initialization.

3. Classifier Co-Decomposition for Model Compression: Classifier co-decomposition isolates shared structure across the collection of class-specific bilinear classifiers.
3. Classifier Co-Decomposition for Model Compression: The factorization uses a shared projection matrix P and compact per-class classifiers V_k in the reduced feature space.P reduces dimensionality from c to m < c, while V_k is the lower-dimensional classifier for class k.
3. Classifier Co-Decomposition for Model Compression: The optimal P spans the singular-vector subspace associated with the largest m singular values of the concatenated classifier parameters.
3. Classifier Co-Decomposition for Model Compression: SVD of the full classifier parameters obtains P and the V_k terms after imposing an orthogonality constraint on P.
3. Classifier Co-Decomposition for Model Compression: The architecture avoids separately learning and factorizing full classifiers by implementing P as a 1 × 1 convolution followed by the new bilinear classifier layer.PCA on base-network feature activations provides an initialization for the projection subspace.

4. Analysis of Computational Efficiency

The analysis compares two low-rank bilinear variants with full and compact bilinear models in feature dimension, parameter memory, and computational complexity. The proposed model is highly compact, with feature computation close to Tensor Sketch but somewhat higher classification cost.

4. Analysis of Computational Efficiency: The evaluation compares LRBP variants against the full bilinear model and compact Random Maclaurin and Tensor Sketch models.
4. Analysis of Computational Efficiency: LRBP-I uses Frobenius-norm scoring and is preferred when hw < m, whereas LRBP-II explicitly pools reduced features and is advantageous when hw > m.
4. Analysis of Computational Efficiency: On CUB200-2011, the model uses P ∈ R^512×100 with m = 100 and classifier rank r = 8 while matching or exceeding compact-model performance.The comparison uses K = 200 classes and Tensor Sketch dimension d = 8,192 as a conventional compact-model setting.
4. Analysis of Computational Efficiency: 100 times smaller than the full bilinear model and 10 times smaller than Tensor Sketch, the proposed model has comparable feature-computation complexity to Tensor Sketch.Its feature complexity is O(hwmc + hwm^2), compared with Tensor Sketch complexity O(hw(c + d log(d))).

5. Experiment Evaluation

The evaluation compares compact bilinear configurations, hyperparameter choices, and qualitative responses across fine-grained benchmarks. The proposed model uses PCA initialization, end-to-end training, and low-rank settings that retain near-maximum accuracy while substantially reducing model size.

Implementation details: The implementation uses VGG16 with fully connected layers removed, followed by co-decomposition, normalization, and bilinear classifier layers.PCA initializes the projection, and the network is fine-tuned end to end.
Implementation details: Sign square root normalization on conv5_3 feature maps improves performance, while adding ℓ2 normalization in the first configuration harms performance.The result is reported as being on par with other bilinear pooling methods for the first configuration.
Hyperparameter configuration: Rank r and reduced dimension m define the architecture, and experiments vary both on CUB-200-2011 without using part annotations or masks.The quick study decomposes full-rank classifiers, applies co-decomposition, and does not fine-tune the resulting models.
Hyperparameter configuration: r = 8 and m = 100 achieve near-maximum CUB-200 classification accuracy while compressing model parameters by a factor of 100 over the full model.Reconstruction fidelity measured by peak signal-to-noise ratio is reported as a useful guide to performance before fine-tuning.
Benchmark setup: The experiments evaluate four fine-grained benchmark datasets using category labels without part or bounding-box annotations.The datasets are CUB-200-2011, Aircrafts, Cars, and DTD.
Quantitative benchmarking: Across the benchmark comparison, bilinear methods achieve similar classification accuracy, while the proposed model is substantially more compact than other bilinear-feature methods.The model is reported to achieve state-of-the-art performance on these datasets using only category labels.
Qualitative visualization: The qualitative visualizations indicate that the model focuses on birds and discriminative parts while ignoring cluttered backgrounds.Highlighted regions change with bird size and context and correspond to distinctive species-specific parts.

6. Conclusion

The conclusion presents a compact low-rank classifier that leverages bilinear pooling without explicitly computing high-dimensional bilinear features. Co-decomposition further reduces computation and storage, while the final model achieves state-of-the-art performance with category labels alone.

Conclusion: The low-rank classifier avoids explicit computation of high-dimensional bilinear pooled features and supports fast test-time evaluation.Hard low-rank constraints reduce parameter degrees of freedom.
Conclusion: Co-decomposition projects features into a shared subspace and further reduces computation and parameter storage.The final model can be initialized with PCA followed by end-to-end fine-tuning.
Conclusion: One to two orders of magnitude smaller than existing approaches, the final classifier achieves state-of-the-art performance on several public fine-grained datasets using only category labels.The reported setting excludes keypoint and part annotations.
Conclusion: The authors identify weakly supervised web-scale training, multiple feature modalities, and mobile-device deployment as future directions.