Source-linked AI summary
Analyzing the Performance of Multilayer Neural Networks for Object Recognition
Pulkit Agrawal, Ross Girshick, Jitendra Malik
TL;DR
CNN features were rapidly becoming alternatives to engineered representations, yet their learned visual structure and transfer behavior remained poorly understood. The paper empirically examines CNN pre-training, fine-tuning, feature coding, and spatial properties across classification and detection tasks, finding that pre-training improves performance and learned codes are mostly distributed.
Problem
CNNs had strong recognition results and appeared poised to replace SIFT and HOG, but the visual features learned by large CNNs were much less understood.
Method
The paper empirically probes CNN pre-training, fine-tuning, feature selectivity, representation structure, and spatial properties across three classification and detection datasets.
Results
Supervised pre-training significantly improves performance, longer pre-training is better, and CNN features are mostly distributed rather than grandmother-cell-like.
Takeaways & Limitations
CNNs can be trained from scratch with modest data, but ImageNet pre-training yields a general and portable representation and supports transfer to other datasets and tasks.
Takeaways & Limitations
An earlier ECCV’14 version incorrectly described the SUN-CLS accuracy metric and reported corrected numbers here.
Abstract
from arXiv · showhide
In the last two years, convolutional neural networks (CNNs) have achieved an impressive suite of results on standard recognition datasets and tasks. CNN-based features seem poised to quickly replace engineered representations, such as SIFT and HOG. However, compared to SIFT and HOG, we understand much less about the nature of the features learned by large CNNs. In this paper, we experimentally probe several aspects of CNN feature learning in an attempt to help practitioners gain useful, evidence-backed intuitions about how to apply CNNs to computer vision problems.
1 Introduction
The paper empirically studies what CNN features encode and how pre-training, feature structure, and spatial information affect their use across vision tasks. It finds that CNNs can learn from modest detection data, while ImageNet pre-training remains beneficial and learned representations are mostly distributed.
- CNNs were expected to replace engineered features, but their learned visual representations were less understood than SIFT and HOG.
- The paper empirically investigates pre-training, fine-tuning, feature representations, and spatial location or magnitude across classification and detection tasks.
- 37k detection bounding boxes can train R-CNN from random initialization, but supervised pre-training still substantially improves detection performance.
- Longer ImageNet pre-training improves transfer performance with diminishing returns and does not increase generalization error on evaluated datasets.
- Intermediate CNN layers contain a small number of grandmother-cell-like features, while most discriminative information forms a distributed code.
- Feature spatial location is critical for object detection but matters little for image classification, whereas feature magnitude is largely unimportant.
2 Experimental setup
The experiments evaluate CNN behavior on standard classification and detection datasets using a single CNN architecture, with specified metrics and training protocols. The setup also includes an enlarged detection dataset and notes a corrected SUN-CLS accuracy metric.
- The study evaluates CNNs on PASCAL-CLS, SUN-CLS, PASCAL-DET, and PASCAL-DET-GT using image classification and object detection tasks.
- Image classification.: PASCAL-CLS contains 5k training images, 5k testing images, and 20 object classes, while SUN-CLS contains approximately 108k images and 397 classes.
- Image classification.: SUN-CLS uses a random 50%/10%/40% train, validation, and test split rather than the dataset’s 10 standard subsets.
- Object detection.: PASCAL-DET trains on VOC 2007 trainval and tests on VOC 2007 test, reporting standard AP and mAP metrics.
- Object detection.: PASCAL-DET+DATA unions VOC 2007 trainval with VOC 2012 trainval, providing approximately 37k labeled bounding boxes, roughly three times PASCAL-DET.
- Object detection.: SUN-CLS results in this version correct an earlier accuracy-metric description error by using class-averaged accuracy instead of overall accuracy.
- All experiments use one Caffe implementation of the Krizhevsky et al. CNN architecture, with pre-training followed by optional fine-tuning.
3 The effects of fine-tuning and pre-training on CNN performance and parameters
Fine-tuning and pre-training substantially affect CNN performance, while their effects depend on data availability and network depth. The experiments also show that higher layers adapt more selectively and that longer pre-training improves transfer performance.
- Detection and classification performance: 40.7% mAP is achievable from scratch on VOC 2007 trainval, but ImageNet pre-training with fine-tuning reaches 54.1% mAP.Adding VOC 2012 data raises scratch performance to 52.3% mAP, nearly matching the pre-trained and fine-tuned result.
- Detection and classification performance: 59.2% mAP is obtained by fine-tuning with additional detection data, 5 percentage points above the prior best result without bounding-box regression.The result indicates that performance was not saturated with the earlier detection training data.
- Detection and classification performance: SUN image classification shows the same pattern: training from scratch is reasonable, but ImageNet initialization followed by fine-tuning performs significantly better.
- Fine-tuning and network parameters: Class selectivity increases from layer 1 to layer 7, while limited-data fine-tuning significantly changes entropy only in fc-6 and fc-7.The analysis measures selectivity using entropy of thresholded filter activations; lower entropy means greater class selectivity.
- Fine-tuning and network parameters: With small amounts of data, fine-tuning mainly rewires the fully connected layers, whereas additional data makes fine-tuning all network parameters substantially more beneficial.
- Pre-training duration: 15k and 50k ImageNet iterations bring all layers close to 80% and 90% of final PASCAL-CLS performance, and longer pre-training improves transfer performance.The first convolutional layer converges before higher layers, which take progressively longer.
4 Are there grandmother cells in CNNs?
The analysis finds only a few GMC-like conv-5 filters, while most classes rely on distributed representations involving multiple filters. Filter overlap is generally low, though related classes share more filters.
- Finding GMC filters: A GMC filter is defined as one with high average precision for classifying a target class against all other classes.High precision and high recall operationalize firing strongly on all target instances and not on other classes.
- Finding GMC filters: GMC-like filters appear for only a few classes, including bicycle, person, cars, and cats.Filters were ranked by average precision for classifying PASCAL-DET-GT bounding-box responses.
- Distributed representations: Around 30 to 40 conv-5 filters are required for most classes to reach at least 90% of full performance, indicating a distributed code.Persons, cars, and cats require relatively few filters, whereas most classes need substantially more.
- Distributed representations: After fine-tuning, slightly fewer filters are needed to reach performance levels similar to those of a pre-trained network.This comparison concerns nested conv-5 filter subsets selected for class-specific linear SVMs.
5 Untangling feature magnitude and location
The ablations show that feature magnitude contributes little to classification and detection performance, whereas spatial information becomes important for detection but less important in higher-layer classification features.
- Magnitude: Binarizing filter responses causes a negligible performance drop for both classification and detection.The transformation preserves which filters fired and where while discarding response magnitude.
- Magnitude: PASCAL-CLS performance in fully connected layers is nearly identical before and after binarization.The result indicates that sparse binary codes arise without an additional encoding procedure.
- Spatial location: For classification, spatial ablation affects conv-1 strongly, but the difference decreases in higher layers and conv-5 sp-max approaches original performance.This indicates that classification-relevant information is largely encoded in filter activations rather than their spatial pattern at higher layers.
- Spatial location: Spatial information remains important for detection because sp-max causes a large performance drop.Precise localization requires spatial information even when classification performance is relatively insensitive to it.
6 Conclusion
The paper empirically studies CNN pre-training, feature selectivity, distributed coding, and ablations of spatial and magnitude information. It finds that pre-training improves performance, representations are mostly distributed, and magnitude is largely unimportant while spatial location matters for detection.
- Conclusion: The study investigates pre-training and fine-tuning behavior, GMC-like filters, distributed feature coding, and the roles of spatial location and magnitude.Experiments span three classification and detection datasets and tasks.
- Conclusion: Pre-training significantly improves performance, and longer pre-training yields better results with diminishing returns.The conclusion also states that the network can be trained from scratch with a surprisingly modest amount of data.
- Conclusion: Most learned features form a distributed code, although a small number are grandmother-cell-like.Several features must fire in concert to discriminate between classes effectively.
Appendix: estimating a filter’s discriminative capacity
The appendix defines a threshold-based entropy measure for estimating a filter’s discriminative capacity. It summarizes how class selectivity changes across response thresholds.
- Response representation: Each image produces a p × p heat map for a filter, whose scores are paired with class labels after vectorization.The resulting score and label vectors each have length p^2.
- Entropy curve: At threshold τ, class entropy is computed from the normalized histogram of labels associated with scores at least τ.Lower entropy means the filter is more class-selective above that threshold.
- Entropy curve: The entropy curve traces class entropy across thresholds, and its area under the curve summarizes discriminative capacity.The area under the entropy curve aggregates selectivity over all response thresholds.
1 Effect of fine-tuning on CNN parameters
The paper examines how fine-tuning changes CNN parameters and uses filter-response entropy metrics to assess those changes.
- Filter-response entropy metrics are used to determine how fine-tuning affects CNN parameters.The analysis collects filter responses and evaluates their discriminative capacity.
Label Entropy.
Class entropy measures a filter’s discriminative capacity across score thresholds, with lower entropy indicating stronger class selectivity.
- For threshold τ, class entropy is computed from the normalized histogram of class labels associated with scores ≥τ.Varying τ produces an entropy curve, whose area under the curve summarizes class entropy across thresholds.
Weighted Label Entropy.
Weighted and Spatial-Max Label Entropy compare filter selectivity before and after fine-tuning, showing larger changes in upper layers and distributed fc-7 representations.
- Weighted Label Entropy.: Weighted Label Entropy uses score sums rather than label counts when constructing the class-label histogram.The scores are nonnegative because they come from rectified linear units.
- Weighted Label Entropy.: Spatial-Max Label Entropy associates each image’s max-pooled filter response with its class label before computing filter AuE.This produces one score and one class label per image.
- Weighted Label Entropy.: Filters are sorted by increasing AuE, and cumulative AuE measures the selectivity of the top i most discriminative filters.The difference between pre- and post-fine-tuning cumulative values measures the change in class selectivity.
- Weighted Label Entropy.: A lower MCAuE indicates more selective filters, and comparing filter fractions assesses selectivity among the most selective filters.Table 9 compares the top 10%, 25%, 50%, and 100% of filters.
- Weighted Label Entropy.: Fine-tuning minimally changes layers 1–4 under Weighted and Spatial-Max Label Entropy, while higher layers become substantially more discriminative.Under Label Entropy, layers 1–5 change negligibly and layers 6–7 become much more discriminative.
- Weighted Label Entropy.: Most fc-7 classes reach 50% performance with one filter, but substantially more filters are required for 90% performance.The representations are distributed across many filters for a large number of classes.