Source-linked AI summary
Learning Deep Features for Discriminative Localization
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, Antonio Torralba
TL;DR
CNNs trained only with image-level labels can localize objects, but it remained unclear whether they could identify discriminative regions across broader tasks. The paper introduces Class Activation Mapping using global average pooling, achieving 37.1% top-5 error for weakly supervised ILSVRC localization, close to AlexNet’s 34.2% fully supervised result, while generalizing to other recognition tasks.
Problem
The paper addresses whether image-level-trained CNNs can identify discriminative image regions beyond object localization across varied visual recognition tasks.
Method
Class Activation Mapping projects output-layer weights onto convolutional feature maps in a global-average-pooling CNN to highlight category-relevant image regions.
Results
37.1% top-5 error was achieved for weakly supervised ILSVRC localization, and the localizable features transferred across classification, localization, and concept-discovery tasks.
Takeaways & Limitations
CAM provides generic localizable deep features for visualizing discriminative object parts and understanding the regions CNNs use for recognition.
Takeaways & Limitations
The approach assumes setting the softmax bias to 0 has little to no impact on classification performance.
Abstract
from arXiv · showhide
In this work, we revisit the global average pooling layer proposed in [13], and shed light on how it explicitly enables the convolutional neural network to have remarkable localization ability despite being trained on image-level labels. While this technique was previously proposed as a means for regularizing training, we find that it actually builds a generic localizable deep representation that can be applied to a variety of tasks. Despite the apparent simplicity of global average pooling, we are able to achieve 37.1% top-5 error for object localization on ILSVRC 2014, which is remarkably close to the 34.2% top-5 error achieved by a fully supervised CNN approach. We demonstrate that our network is able to localize the discriminative image regions on a variety of tasks despite not being trained for them
1. Introduction
The paper shows that an appropriately modified global average pooling architecture preserves CNN localization ability through the final layer, enabling class-specific discriminative localization from image-level labels. It achieves competitive weakly supervised object localization and transfers localizable deep features across recognition tasks.
- Core contribution: A modified global average pooling layer lets a classification-trained CNN identify discriminative image regions in a single forward pass.Global average pooling extends beyond structural regularization, allowing localization to persist until the final layer.
- Transfer across tasks: CNNs trained for object categorization localize action-relevant objects that humans interact with, rather than the humans themselves.This demonstrates localization beyond ordinary object-localization targets.
- Results: 37.1% top-5 test error is achieved for weakly supervised object localization on ILSVRC, versus 34.2% for fully supervised AlexNet.The result is described as close despite the apparent simplicity of the approach.
- Discriminative localization: The architecture generalizes image-level-label localization beyond objects to identifying the regions used for discrimination.The paper presents this as a broader capability enabled by the right architecture.
- Relation to prior work: Unlike global max pooling, global average pooling is argued to encourage identifying an object’s complete extent rather than only a boundary point.The paper also emphasizes that accurate discriminative localization, rather than global average pooling itself, is its novel observation.
2. Class Activation Mapping
Class activation mapping uses global average pooling to map class scores back onto the last convolutional feature maps, revealing class-specific discriminative image regions. The approach supports localization across tasks and, compared with global max pooling, produces better localization despite similar classification performance.
- CAM procedure: CAMs identify the discriminative image regions used by a CNN to recognize a particular category.The predicted class score is mapped back to the previous convolutional layer to generate class-specific activation maps.
- CAM procedure: A GAP-based architecture averages each last-layer feature map spatially, then combines these averages through a fully connected output layer.The same connectivity structure can produce categorical or other outputs.
- CAM procedure: Each CAM is a weighted linear sum of visual-pattern activations across spatial locations, and upsampling it identifies regions most relevant to the category.The map value at each spatial location indicates the importance of that activation for classification.
- CAM behavior: CAMs highlight different discriminative regions for different categories, even within the same image.Examples include distinct regions activated by categories such as dome and palace.
- GAP versus GMP: GAP outperforms GMP for localization while achieving similar classification performance on ILSVRC.GAP encourages identifying the object’s extent, whereas GMP tends to focus on a single discriminative part.
3. Weakly-supervised Object Localization
The section evaluates CAM-based weakly supervised localization on ILSVRC 2014 using CNNs modified with global average pooling. These networks preserve classification performance closely while achieving strong localization without annotated bounding boxes.
- 3.1 Experimental Setup: AlexNet, VGGnet, and GoogLeNet were converted by replacing their pre-output fully connected layers with GAP and a fully connected softmax layer.The resulting networks were fine-tuned on 1.3M ILSVRC training images for 1000-way classification.
- 3.1 Experimental Setup: Higher mapping resolution improved localization, so layers after selected convolutional stages were removed and a 3 × 3 convolutional layer with 1024 units was added before GAP.VGGnet and GoogLeNet used a 14 × 14 mapping resolution after these modifications.
- 3.2 Results: Classification performance usually dropped only 1–2% after removing layers, while AlexNet*-GAP restored performance to a level comparable to AlexNet.AlexNet was most affected, motivating the addition of two convolutional layers before GAP.
- 3.2 Results: CAM bounding boxes were generated by thresholding regions above 20% of the heatmap maximum and selecting the largest connected component for each top-5 predicted class.Localization used the same top-1 and top-5 error metrics as ILSVRC and was evaluated on validation and test sets.
- 3.2 Results: 43% top-5 localization error was achieved by GoogLeNet-GAP on the ILSVRC validation set, outperforming the baselines without any annotated bounding-box training.CAM also significantly outperformed backpropagation-based localization and GoogLeNet on localization.
4. Deep Features for Generic Localization
GoogLeNet-GAP learns generic visual features that are competitive across scene and object benchmarks while retaining class activation maps that highlight discriminative regions. These localizable features support fine-grained bird recognition, visual pattern discovery, text detection, and visual question answering without task-specific bounding-box training.
- Generic features: GoogLeNet-GAP features are competitive generic visual representations across eight scene, object, action, and event classification benchmarks.They significantly outperform AlexNet fc7 features, while performing similarly to GoogLeNet despite having fewer convolutional layers.
- Generic localization: GoogLeNet-GAP class activation maps highlight the most discriminative regions across datasets, demonstrating generic localization beyond the training task.The network was trained for object recognition, but its localization maps remain informative for other datasets and tasks.
- Fine-grained recognition: 70.5% accuracy is achieved on CUB-200-2011 when bounding box annotations are used, compared with 63.0% using the full image without annotations.Localized bird crops can be extracted from CAMs and used to improve fine-grained classification because category distinctions are subtle.
- Fine-grained recognition: 41.0% of bird images are accurately localized at 0.5 IoU, compared with 5.5% chance performance.This quantitatively validates GoogLeNet-GAP’s ability to localize birds in the fine-grained recognition setting.
- Pattern discovery and applications: CAMs identify informative regions for abstract concepts and text without bounding-box annotations, and support visual question answering with 55.89% test-standard accuracy.The method localizes concept regions from caption-based weak labels, accurately highlights text in StreetView images, and highlights regions relevant to predicted answers.
5. Visualizing Class-Specific Units
The section identifies class-specific convolutional units as visual concept detectors and uses GAP with ranked softmax weights to determine their importance for different classes. Applied to AlexNet*-GAP, this procedure reveals discriminative object parts and scene-related units across recognition datasets.
- Class-Specific Units: Convolutional units detect visual concepts ranging from textures and materials to objects and scenes, becoming increasingly discriminative in deeper layers.Fully connected layers can make the importance of individual units for category identification difficult to determine.
- Class-Specific Units: GAP and ranked softmax weights provide a way to identify and rank units according to their importance for a given class.The method addresses the difficulty of interpreting unit importance in networks containing fully connected layers.
- Class-Specific Units: Fig. 13 visualizes class-specific units in AlexNet*-GAP trained on ILSVRC for object recognition and Places Database for scene recognition.The procedure estimates receptive fields, segments top activation images from the final convolutional layer, and ranks units using softmax weights.
6. Conclusion
The paper proposes Class Activation Mapping (CAM), enabling classification-trained CNNs with global average pooling to perform object localization without bounding-box annotations. CAM also generalizes across visual recognition tasks as a generic localizable deep feature representation.
- Conclusion: Class Activation Mapping enables classification-trained CNNs with global average pooling to perform object localization without bounding-box annotations.The approach was evaluated on weakly supervised object localization using the ILSVRC benchmark.
- Conclusion: CAM visualizes predicted class scores on a given image, exposing the image regions contributing to a CNN’s classification.
- Conclusion: The localization technique generalizes to other visual recognition tasks by producing generic localizable deep features.These features can help researchers understand the basis of discrimination used by CNNs for their tasks.