Source-linked AI summary
Bilinear CNNs for Fine-grained Visual Recognition
Tsung-Yu Lin, Aruni RoyChowdhury, Subhransu Maji
TL;DR
Fine-grained recognition needs localized, pose- and location-invariant distinctions, while prior deep texture representations lacked end-to-end feature learning. The paper introduces B-CNNs, which pool outer products of two CNN feature streams, and reports strong results across recognition tasks with efficient, trainable models.
Problem
Prior CNN-based texture representations were effective for fine-grained recognition but had not matched part-based methods and were not trained end to end.
Method
B-CNNs combine features from two CNNs through location-wise outer products and pooling, forming an orderless representation that can be trained end to end.
Results
B-CNNs achieve the best accuracy across the reported models on bird species classification, with the best model obtaining 84.1% accuracy.
Takeaways & Limitations
B-CNN variants are effective across fine-grained, texture, and scene recognition tasks and support training from scratch on ImageNet.
Takeaways & Limitations
End-to-end training can be slowed by GPU implementation issues involving the bilinear layer, so skipping that step may be necessary for efficient fine-tuning.
Abstract
from arXiv · showhide
We present a simple and effective architecture for fine-grained visual recognition called Bilinear Convolutional Neural Networks (B-CNNs). These networks represent an image as a pooled outer product of features derived from two CNNs and capture localized feature interactions in a translationally invariant manner. B-CNNs belong to the class of orderless texture representations but unlike prior work they can be trained in an end-to-end manner. Our most accurate model obtains 84.1%, 79.4%, 86.9% and 91.3% per-image accuracy on the Caltech-UCSD birds [67], NABirds [64], FGVC aircraft [42], and Stanford cars [33] dataset respectively and runs at 30 frames-per-second on a NVIDIA Titan X GPU. We then present a systematic analysis of these networks and show that (1) the bilinear features are highly redundant and can be reduced by an order of magnitude in size without significant loss in accuracy, (2) are also effective for other image classification tasks such as texture and scene recognition, and (3) can be trained from scratch on the ImageNet dataset offering consistent improvements over the baseline architecture. Finally, we present visualizations of these models on various datasets using top activations of neural units and gradient-based inversion techniques. The source code for the complete system is available at http://vis-www.cs.umass.edu/bcnn.
1 INTRODUCTION
Fine-grained recognition requires distinguishing subordinate categories using localized attributes while remaining invariant to object pose and location. B-CNNs address limitations of deep texture representations by making their feature extraction trainable end to end.
- Fine-grained recognition classifies instances within subordinate categories such as bird species, car models, and dog breeds.
- Localized attributes, including bill patterns and feather colors, must be recognized despite variation in object pose and image location.
- Part-based models localize parts and extract features conditioned on their detected locations, while texture representations aggregate local features in an orderless manner.
- CNN-based texture representations have been effective for fine-grained recognition but have not matched part-based approaches.
- B-CNNs express texture representations as pooled outer products of CNN features, followed by specialized bilinear and pooling layers for end-to-end training.
- The manuscript analyzes B-CNNs across fine-grained, material, texture, and scene recognition, including dimensionality trade-offs and ImageNet training from scratch.
2 RELATED WORK
Related work spans fine-grained part-based methods, deep texture and second-order representations, compact bilinear features, and architectures for multiplicative interactions. B-CNNs are positioned as end-to-end trainable recognition models distinct from texture generation approaches.
- Fine-grained methods combine CNN features with part detectors or weakly supervised segmentation to improve recognition.
- Deep architectures such as VGG, Inception, residual, and spatial transformer networks have improved fine-grained recognition.
- Additional training data and boosting have been reported to improve deep-network and B-CNN accuracy, respectively.
- Texture research includes second-order filter-bank statistics and representations such as Fisher Vectors and O2P for image classification and segmentation.
- Orderless CNN-feature aggregation improves texture, scene, and fine-grained recognition, but prior filter banks were not trained end to end.
- The Gram matrix equals pooled bilinear features when both features are identical, but this work emphasizes recognition rather than texture synthesis or style transfer.
- Naively learning pairwise interactions over O(n) features requires O(n^3) parameters, whereas an explicit outer product has no parameters.
- Low-dimensional bilinear features reduce memory overhead, including reduction from 512×512 to 512×64 dimensions by projecting one feature.
3 B-CNNS FOR IMAGE CLASSIFICATION
The paper introduces B-CNNs for image classification and shows that widely used texture representations can be formulated within this architecture.
- B-CNNs provide an image-classification architecture that represents various widely used texture representations within a common formulation.
3.1 The B-CNN architecture
A B-CNN combines feature functions from two CNNs through location-wise outer products, pools the resulting interactions into a global representation, and classifies it. The architecture supports normalization and end-to-end gradient-based training.
- A B-CNN consists of CNN-based feature functions fA and fB, a pooling function P, and a classification function C.
- The image passes through CNNs A and B, whose outputs at each location are combined by an outer product and average pooled into bilinear features.
- Feature outputs are combined at each location using a matrix outer product, with compatible feature functions sharing feature dimension K.
- Pooling aggregates bilinear combinations across locations into a global representation, making the result orderless and preserving pairwise feature interactions.
- CNN feature functions produce location-indexed features and may use shared or independent feed-forward computations.
- Bilinear features undergo signed square-root and ℓ2 normalization before classification with logistic regression or a linear SVM.
- The bilinear architecture permits back-propagation through the classification loss, enabling end-to-end training when feature gradients are efficiently computed.
3.2 Relation to classical texture representations
B-CNNs express several classical orderless texture representations as pooled outer products and adapt them into end-to-end trainable formulations. This framework connects representations including VLAD, FV, BoVW, and O2P to CNN-based feature learning while retaining localized appearance aggregation.
- Orderless texture representations aggregate local image features to capture translation-invariant texture properties through choices of features, encoding, pooling, and normalization.The feature function is decomposed into location- and image-dependent processing followed by additional nonlinearities.
- Encoder and appearance interactions act as joint statistics that can describe object parts regardless of their image location, supporting fine-grained recognition.The encoders can behave like part detectors, while the appearance features capture the associated local visual content.
- VLAD, FV, BoVW, and O2P can be written as bilinear outer-product models, with O2P corresponding to symmetric bilinear pooling when both feature functions are identical.VLAD and FV use encoder-dependent features, while O2P uses identical SIFT features on both sides.
- End-to-end trainable approximations replace hard assignments with differentiable soft assignments and simplify other descriptor-specific computations.The formulations include soft-assignment replacements for VLAD and BoVW, while NetFV adds element-wise second-order statistics but discards covariance and prior terms from the true FV model.
- Skipping O2P’s log-Euclidean mapping enables efficient fine-tuning, while B-CNN models improve accuracy across varied fine-grained datasets.The mapping requires an expensive eigenvalue decomposition without efficient GPU implementation; the reported B-CNN models are trained using back-propagation-based fine-tuning.
- B-CNNs pool outer products of features from two CNNs, and their feature functions may share no computations, share some computations, or share all computations.Examples include separate VGG-M and VGG-D networks, partially shared NetVLAD-style models, and fully shared VGG-M networks.
4.2 Fine-grained recognition
B-CNNs achieve strong fine-grained recognition performance across bird, aircraft, and car datasets, generally outperforming the compared texture-representation baselines without requiring part or bounding-box annotations.
- Experimental setting: B-CNNs use only category labels for training on CUB and NABirds, although these datasets provide part and bounding-box annotations.The datasets contain challenging variation in pose, viewpoint, clutter, and fine-grained category differences.
- Bird species classification: 84.1% accuracy is achieved by the best B-CNN model on CUB, exceeding NetVLAD's 81.9%.The best model combines VGG-M and VGG-D features.
- Bird species classification: 79.4% accuracy on NABirds exceeds the 63.7% obtained by a fine-tuned VGG-D network.The B-CNN uses VGG-M and VGG-D features and reports softmax predictions without the SVM training step.
- Aircraft variant classification: 86.9% accuracy is obtained by a B-CNN on FGVC-aircraft, compared with 81.4% for NetVLAD.The aircraft images are resized to 512 × 512 and centrally cropped to 448 × 448.
- Comparison to baselines: Table 2 compares representations with and without domain-specific fine-tuning, and directly fine-tuning approximate models performs better than indirect fine-tuning.B-CNN models achieve the best accuracy among the compared texture representations.
- Car model classification: 91.3% accuracy is obtained by the best B-CNN on Stanford cars, compared with 88.6% for NetVLAD.The result uses the VGG-D + VGG-M model.
4.3 Texture and scene recognition
B-CNNs are evaluated on texture and indoor-scene datasets against Fisher Vector pooling of CNN features, with multi-scale inputs consistently improving accuracy and fine-tuning helping mainly on MIT indoor scenes.
- Datasets and protocol: The evaluation covers DTD, FMD, KTH-T2b, and MIT indoor datasets with mean per-class accuracy reported for FV and B-CNN representations.The datasets span Internet-collected textures, controlled material images, and weakly structured indoor scenes.
- Scale comparison: Multi-scale inputs consistently improve accuracy for both B-CNN and Fisher Vector representations across the evaluated texture and scene datasets.Table 3 compares single-scale and multiple-scale inputs built from VGG-D relu5 3 features.
- Fine-tuning: 72.8% → 73.8% is the reported MIT indoor improvement from fine-tuning B-CNNs, while improvements on the other datasets are negligible.The authors attribute the small changes on the other datasets as likely related to their relatively small size.
- Representation comparison: B-CNN and Fisher Vector performance is identical across input scales in the reported comparison.The comparison uses VGG-D features without fine-tuning.
5 ANALYSIS OF BILINEAR CNNS
The analysis studies dimensionality reduction, ImageNet training, and visualizations for B-CNNs and related texture representations. It finds that compact bilinear features retain accuracy, B-CNNs outperform comparable representations at matched dimensions, and learned units capture localized attributes.
- 5.1 Dimensionality reduction: 262K-dimensional B-CNN descriptors are highly redundant, motivating dimensionality reduction without significant classification loss.Earlier work reported order-of-magnitude reductions in feature size without loss in classification performance.
- 5.1 Dimensionality reduction: Reducing one feature with PCA before the outer product works well, whereas projecting both features causes a significant accuracy reduction.The one-sided projection can be initialized with PCA and trained jointly with the classification layers.
- 5.1 Dimensionality reduction: More than 32 cluster centers does not improve NetVLAD or NetFV performance on the evaluated fine-grained datasets.Figure 5 varies the number of cluster centers before and after fine-tuning; NetFV uses twice the feature dimension of NetVLAD at equal cluster counts.
- 5.1 Dimensionality reduction: 16× dimensionality reduction to 32×512 yields less than 1% accuracy loss for B-CNNs on birds and aircrafts.At the same feature size, NetVLAD with 32 components is about 3-4% less accurate.
- 5.1 Dimensionality reduction: Projected B-CNNs outperform NetVLAD and NetFV at a fixed dimensionality budget, while PCA is 1.5× faster than Compact Bilinear Pooling.PCA is slightly less accurate than Compact Bilinear Pooling, but uses dense matrix multiplication instead of Fourier transforms and inverses.
- 5.2 Training B-CNNs on ImageNet LSVRC: B-CNN achieves 38.7% top1 error with f1 augmentation, outperforming VGG-M's 39.6% top1 error with f25 augmentation.The experiments train both networks from scratch on ImageNet and evaluate single center-cropped images; the authors report robustness to translation and greater effectiveness of explicit translation invariance.
6 CONCLUSION
The paper concludes that B-CNNs provide efficient, end-to-end trainable orderless representations based on second-order CNN statistics. Across fine-grained, texture, and scene recognition, compact versions remain effective and learned units reflect localized visual attributes.
- 6 CONCLUSION: B-CNNs aggregate second-order CNN activation statistics into an orderless image representation that supports end-to-end training and fine-tuning.The models process 448×448 images at 30-100 FPS on an NVIDIA Titan X GPU.
- 6 CONCLUSION: B-CNN variants are effective across fine-grained, texture, and scene recognition tasks, while their dimensions can usually be reduced by an order of magnitude without significant accuracy loss.The conclusion frames this as an accuracy-memory trade-off across exact and approximate outer-product representations.
- 6 CONCLUSION: Visualization indicates that B-CNN units represent objects as textures and correlate with localized attributes useful for fine-grained recognition.The conclusion connects these localized attributes with the model's learned representation of objects.