Source-linked AI summary
Self-taught Object Localization with Deep Networks
Loris Bazzani, Alessandro Bergamo, Dragomir Anguelov, Lorenzo Torresani
TL;DR
The paper targets object localization without manually annotated bounding boxes, which are costly and subjective. It analyzes recognition-score changes when image regions are masked and clusters regions into localization hypotheses. On ILSVRC-2012, the method outperformed objectness approaches for small proposal budgets, while automatically generated hypotheses supported detector training with results close to manual-box training.
Problem
Object detectors require costly, time-consuming, subjective, and noisy manually selected bounding boxes, whereas image class labels are easier to obtain.
Method
The method masks image regions, measures recognition-score drops, and embeds these signals in hierarchical clustering to generate likely object-containing subwindows.
Results
23.4% relative top-1 recall improvement over BING was achieved on ILSVRC-2012, and STL-trained detectors performed nearly comparably to detectors trained on manual boxes.
Takeaways & Limitations
Automatically generated localization hypotheses can support object-detector training without additional human supervision.
Takeaways & Limitations
Localization can fail when multiple same-class instances or contextual correlations keep classification active after masking the target object.
Abstract
from arXiv · showhide
This paper introduces self-taught object localization, a novel approach that leverages deep convolutional networks trained for whole-image recognition to localize objects in images without additional human supervision, i.e., without using any ground-truth bounding boxes for training. The key idea is to analyze the change in the recognition scores when artificially masking out different regions of the image. The masking out of a region that includes the object typically causes a significant drop in recognition score. This idea is embedded into an agglomerative clustering technique that generates self-taught localization hypotheses. Our object localization scheme outperforms existing proposal methods in both precision and recall for small number of subwindow proposals (e.g., on ILSVRC-2012 it produces a relative gain of 23.4% over the state-of-the-art for top-1 hypothesis). Furthermore, our experiments show that the annotations automatically-generated by our method can be used to train object detectors yielding recognition results remarkably close to those obtained by training on manually-annotated bounding boxes.
1. Introduction
The paper addresses the costly, subjective bounding-box supervision required for object detection by using whole-image classifiers to generate localization hypotheses without object-location annotations. Its method analyzes recognition-score changes under region masking and combines this signal with hierarchical region grouping.
- Motivation: Object detection localizes objects and is more robust to background clutter than whole-image classification, but requires bounding-box annotations.Manual selections are expensive, time-consuming, subjective, and noisy.
- Motivation: Class labels are easier to obtain at scale, enabling datasets such as ImageNet and transfer of classification models to detection and localization.The passage also identifies transfer to attribute classification.
- Method: Self-taught localization uses a whole-image convolutional network trained with class labels, without object-location information.The method leverages a classifier trained on a large collection of class-labeled examples.
- Method: Masking an object-containing region typically causes a significant recognition-score drop, providing a localization signal.The partial masking operation is propagated through the convolutional network to the classification output.
- Method: Hierarchical clustering merges regions according to relative recognition-score drops to produce likely object-containing subwindows.The method combines bottom-up grouping with top-down discriminative information from the network.
- Results: On ILSVRC-2012, the method improved top-1 recall by 23.4% relative to the state of the art and approached manually supervised detector results.It also achieved state-of-the-art results across proposal counts from 1 to 10^4 and generalized to PASCAL 2007.
2. Related work
Prior deep localization and objectness methods either rely on manually annotated boxes or generate many low-precision proposals. The paper positions self-taught localization as a weakly labeled alternative that uses masking effects for localization.
- Deep localization: Earlier deep localization methods fine-tune networks on ground-truth boxes or regress directly to bounding-box vectors.These approaches require manually annotated bounding boxes.
- Objectness methods: Objectness methods generate high-recall boxes but often produce many false positives, limiting their use as detector-training ground truth.Their purpose is to cover each object with at least one candidate subwindow.
- Network analysis: Prior work examined intermediate representations, semantic units, emergent detectors, and classifier responses to artificial images to understand deep networks.These studies addressed network interpretation rather than the paper's localization method.
- Masking and saliency: Masking input regions had been explored for visualizing learned features, while related work also computed class-specific saliency maps.The paper distinguishes its approach by applying masking to localization and reporting state-of-the-art weakly labeled results.
3. Self-Taught object localization
Self-Taught Localization masks image regions, measures classification-score drops, and uses those signals with visual and spatial cues to generate object-containing proposals. It supports both class-specific and weakly labeled settings through agglomerative region clustering.
- Input mask-out: STL masks rectangular image regions with a learned mean RGB vector before evaluating the deep network.The masking function replaces pixels in region b with the learned mean value of the image channels.
- Classification drops: A masked region is considered discriminative for a class when its removal causes a large positive classification-score difference.The score variation compares the original image with its masked version component-wise, using nonnegative differences.
- Classification drops: With the ground-truth class available, dCL produces class-specific proposals likely to contain instances of that class.This variant is called STLCL and uses the indicator vector for class c.
- Classification drops: Without class labels, STLWL uses the classifier’s top-5 predicted classes to define the weakly labeled drop score.The method relies on estimated labels; the cited network’s top-5 correct-class probability is 82%.
- Agglomerative clustering: Agglomerative clustering begins with K rectangular regions and repeatedly merges the pair with maximum similarity until one whole-image region remains.Generated subwindows are sorted by classification drop, followed by non-maximum suppression for overlaps greater than 50%.
- Agglomerative clustering: The merging similarity combines classification-drop, appearance, size, and spatial-fill terms with uniform weights.The drop term favors regions with similar and substantial classification drops; removing it reduces precision by 8% and recall by 10%.
- Method characterization: STL combines bottom-up grouping cues with top-down class-recognition information to generate class-specific windows without hand-engineered features.Its learned-network similarity differs from the purely bottom-up region merging used by the comparison method.
4. Experiments
Experiments evaluate STL for object proposals, cross-dataset generalization, detector training, and failure cases. STL achieves strong precision and recall with few proposals, generalizes to PASCAL-VOC-2007, and produces useful detector-training annotations.
- Object proposal evaluation: STL outperforms competing proposal methods for small proposal budgets, achieving a 23.4% relative top-1 recall improvement over BING on ILSVRC-2012-LOC.The comparison reports recall and precision on the validation set.
- Object proposal evaluation: Combining STLWL with MCG preserves STL's advantage for 1–10 proposals while achieving competitive recall and precision for 11–10^4 proposals.MCG performs best at larger proposal counts, whereas STLWL is strongest at small counts.
- Cross-dataset generalization: STL generalizes to PASCAL-VOC-2007 despite being trained without images from that dataset, especially in mean precision, and complements EDGEBOXES.The PASCAL categories do not exactly correspond to ILSVRC categories.
- Cross-dataset generalization: Across PASCAL classes, higher maximum correlation with ILSVRC prototypes coincides with higher top-1 recall from STL.Figure 4 plots prototype correlation against STL recall for each PASCAL category.
- Detector training: STLCL+STLWL outperforms all compared detector-training methods, improving mean average precision by 7% over SELSEARCH and 42.2% over BING.The evaluation uses 200 randomly selected ILSVRC-2012-LOC classes.
- Detector training: STLCL+SELSEARCH is within a 19.6% relative performance drop of fully supervised training while using only class labels, and exceeds ground-truth annotations on 41 classes.The class-level comparison reports higher accuracy for STLCL+SELSEARCH on points above the diagonal.
5. Conclusions
The work presents self-taught localization using class-labeled convolutional networks to generate object proposals automatically. Detectors trained on these hypotheses perform nearly comparably to those trained on manually selected bounding boxes.
- Self-taught localization uses convolutional networks trained with image class labels to automatically generate object proposal subwindows.
- Detectors trained on automatically generated localization hypotheses achieve performance nearly comparable to detectors trained on manually selected bounding boxes.
Abstract
The supplementary material provides additional evidence about the proposed method through visualizations of masking effects and comparisons of generated bounding boxes. It also notes that the document is best viewed in color.
- The supplementary material visualizes the mask-out effect through convolutional feature maps and classification-score drops.
- It compares top-1 bounding boxes generated by STLCL and SELSEARCH.
- The document is best viewed in color.
Visualizing the Mask-out Effect
The experiments visualize how masking propagates through convolutional layers and affects classification scores, while also examining failures caused by multiple instances and contextual cues. Sample bounding boxes illustrate the resulting localization behavior.
- Successful mask-out cases show object removal propagating through convolutional feature maps and causing a significant classification-score drop.The mask-out operation corresponds to zeroing feature-map values associated with pixels in the masked region.
- Localization can fail when another instance remains recognizable after one object is masked.For two dogs, masking one produces only a small score drop because the other dog still activates the network.
- Contextual information learned by the network can keep classification scores high even after the object is masked.Examples include background landscapes associated with eagles and co-occurring basketball players associated with basketballs.
- STLCL produces more accurate sample subwindows than SELSEARCH in the reported bounding-box comparisons.