Source-linked AI summary
Object Detectors Emerge in Deep Scene CNNs
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, Antonio Torralba
TL;DR
The paper addresses the difficulty of understanding representations learned by deep CNNs. It analyzes CNNs trained for scene classification and finds that object detectors emerge without object labels, enabling the same network to support scene recognition and object localization in one forward pass.
Problem
The nature of learned representations inside deep CNNs remains unclear, especially how meaningful visual parts should be identified for recognition.
Method
The paper analyzes scene-trained CNN representations using minimal image representations, occlusion-based receptive-field maps, and semantic analysis of internal units.
Results
Object detectors emerge inside CNNs trained for scene classification, with Places-CNN discovering more objects than ImageNet-CNN despite no object-level supervision.
Takeaways & Limitations
A single network can support recognition across edges, textures, objects, and scenes while performing object localization and scene recognition in one forward pass.
Takeaways & Limitations
Some Places-CNN units do not detect objects, so texture-based or part-based representations may also contribute to scene recognition.
Abstract
from arXiv · showhide
With the success of new computational architectures for visual processing, such as convolutional neural networks (CNN) and access to image databases with millions of labeled examples (e.g., ImageNet, Places), the state of the art in computer vision is advancing rapidly. One important factor for continued progress is to understand the representations that are learned by the inner layers of these deep architectures. Here we show that object detectors emerge from training CNNs to perform scene classification. As scenes are composed of objects, the CNN for scene classification automatically discovers meaningful objects detectors, representative of the learned scene categories. With object detectors emerging as a result of learning to recognize scenes, our work demonstrates that the same network can perform both scene recognition and object localization in a single forward-pass, without ever having been explicitly taught the notion of objects.
1 INTRODUCTION
The paper asks what deep CNNs learn internally and studies scene classification as a clearer setting for understanding representations. It shows that object detectors emerge inside scene-trained CNNs without object supervision, supporting localization and scene recognition in one forward pass.
- Motivation: Deep CNNs outperform hand-crafted features, but their learned internal representations remain difficult to interpret.The paper motivates examining why these representations work and how they support transfer to tasks such as object detection.
- Motivation: Prior analyses of CNN representations largely used ImageNet, an object-centric dataset, limiting what could be learned about representations beyond object recognition.Existing work examined unit visualization, feature transfer, and distributed object codes using ImageNet training.
- Motivation: Scene representations are structured by the objects they contain and by their relatively loose spatial configurations, allowing shared object codes across scene categories.Examples include beds, lamps, cabinets, walls, floors, and ceilings in bedrooms, alongside textures and other scene regularities.
- Contribution: Object detectors emerge inside a CNN trained for scene recognition, even more than inside an ImageNet-trained CNN, despite receiving no object supervision.The scene-classification objective naturally discovers many objects relevant to discriminating among scene categories.
- Contribution: A single scene-trained network can recognize edges, textures, objects, and scenes while performing object localization and scene recognition in one forward pass.This differs from approaches requiring repeated network applications, attention, segmentation, or object-level supervision.
2 IMAGENET-CNN AND PLACES-CNN
The paper compares CNNs trained from scratch on object-centric ImageNet and scene-centric Places data. Places-CNN develops stronger scene-related features, with later-layer activation preferences becoming specialized to the training task.
- Training setup: ImageNet-CNN uses 1.3 million images from 1000 object categories and reaches 57.4% top-1 accuracy, while Places-CNN uses 2.4 million images from 205 scene categories and reaches 50.0%.Both networks use the same architecture and are trained from scratch on their respective datasets.
- Representation differences: Places-CNN features outperform ImageNet-CNN features on scene-related recognition tasks.On the same scene-classification test set, Places-CNN achieves 50.0%, whereas ImageNet-CNN with a linear SVM achieves 40.8%.
- Representation differences: The activation-preference experiment ranks the top 100 images per layer from a balanced 200k-image set containing scene-centric and object-centric images.The analysis records each layer’s average activation across spatial locations and compares the preferred image types.
- Representation differences: Early layers prefer similar images in both networks, whereas later layers become specialized to scene or object categorization.At fc7, 78% of the ImageNet-CNN top-100 images come from ImageNet compared with 24% for Places-CNN.
3 UNCOVERING THE CNN REPRESENTATION
The paper probes CNN representations using simplified images, receptive-field estimation, and unit concept annotation. These analyses show that scene-trained networks learn increasingly semantic, object-sensitive representations without object-level supervision.
- 3.1 SIMPLIFYING THE INPUT IMAGES: The simplification procedure removes image segments iteratively, preserving the smallest approximately sufficient information for correct scene classification.At each iteration, the segment causing the smallest decrease in correct classification score is removed until the image is misclassified.
- 3.1 SIMPLIFYING THE INPUT IMAGES: Minimal image representations preserve scene categories while retaining diagnostic objects, such as beds in bedrooms and paintings in art galleries.Bedroom representations retained beds in 87% of cases; art-gallery representations contained paintings in 81% and pictures in 58%.
- 3.2 VISUALIZING THE RECEPTIVE FIELDS OF UNITS AND THEIR ACTIVATION PATTERNS: Empirical receptive fields are estimated by occluding dense grids of image patches in highly activating images and aggregating activation discrepancies.The method uses 11×11 randomized occluders with stride 3, then recenters and averages discrepancy maps across top-ranked images.
- 3.2 VISUALIZING THE RECEPTIVE FIELDS OF UNITS AND THEIR ACTIVATION PATTERNS: Deeper layers have larger receptive fields and more semantically meaningful activation regions, while empirical receptive fields are smaller than theoretical ones.The analysis compares pool1, pool2, conv4, and pool5 units across Places-CNN and ImageNet-CNN.
- 3.3 IDENTIFYING THE SEMANTICS OF INTERNAL UNITS: Across layers, early units respond more to simple elements and colors, whereas later units respond more to objects and scenes.Among units exceeding 75% annotation precision, Places-CNN conv4 and pool5 contain higher ratios of high-level semantics than ImageNet-CNN.
- 3.3 IDENTIFYING THE SEMANTICS OF INTERNAL UNITS: Places-CNN discovers more object-selective units than ImageNet-CNN despite receiving no object-level supervision.The concept distributions show more objects and scenes in Places-CNN, while ImageNet-CNN has more units tuned to simple elements, colors, and object parts.
4 EMERGENCE OF OBJECTS AS THE INTERNAL REPRESENTATION
The analysis asks which object classes emerge in pool5 and finds that Places-CNN units often detect objects, with discovered categories strongly related to scene discrimination. These inner-layer units can also support object localization, although many units do not detect objects.
- Object classes: Pool5 contains multiple units for some object classes, including 15 Places-CNN units that detect buildings.The analysis groups units by apparent object class and examines their strongest activating images and segmentation regions.
- Why objects emerge: The correlation between discovered object frequency and scene-discriminative object frequency is 0.84, exceeding the correlation with database object frequency, which is 0.54.The discriminative-object analysis counts how often each class is most informative for scene categories.
- Scope: Of Places-CNN’s pool5 units, 115 do not detect objects, so texture-based, part-based, or incompletely learned representations may also contribute.The paper therefore does not rule out representations used in combination with object detectors.
- Object localization: A single Places-CNN can support scene recognition and object localization in one forward-pass by interpreting activated inner-layer units.Bounding boxes identify regions where units activate above a threshold within their receptive fields.
- Object localization: Many discovered pool5 units achieve very high segmentation performance, supporting their interpretation as object detectors despite scene-classification training.The segmentation evaluation uses fully annotated SUN images from the same 205 place categories used to train Places-CNN.
5 CONCLUSION
The conclusion is that scene-classification training produces object detectors in inner CNN layers, allowing one network to represent multiple abstraction levels. Only objects informative for particular scene-recognition tasks emerge, motivating investigation of other tasks.
- Conclusion: Scene-classification training produces object detectors and supports recognition at levels including edges, textures, objects, and scenes within one network.The network can provide reliable outputs from inner layers without multiple outputs or networks.
- Conclusion: Object detectors emerge because objects compose scenes, especially those that discriminate among scene categories.The conclusion links the learned inner-layer detectors to the objects informative for scene recognition.
- Future work: Only objects informative for specific scene-recognition tasks emerge, leaving open which other tasks could produce other object classes without object-label supervision.The paper identifies this as a direction for future work.