Source-linked AI summary
LSDA: Large Scale Detection Through Adaptation
Judy Hoffman, Sergio Guadarrama, Eric Tzeng, Ronghang Hu, Jeff Donahue, Ross Girshick, Trevor Darrell, Kate Saenko
TL;DR
Object detection does not scale easily because bounding-box annotations are scarce, while image-level classification labels are abundant. LSDA learns the transformation from classification to detection using categories with both labels, then transfers it to categories without boxes, yielding a 50% relative improvement and supporting large-scale detection.
Problem
Bounding-box annotations cover only a small fraction of categories, whereas image-level labels are much easier to collect at scale.
Method
LSDA treats classification-to-detection transfer as domain adaptation, learning shared and category-specific transformations from categories with both types of labels.
Results
50% relative improvement was achieved over raw classifier weights on object proposal regions, with 16.15% mAP on held-out categories for output adaptation with k = 10.
Takeaways & Limitations
LSDA can produce detectors for categories without bounding-box data and has the potential to extend detection to tens of thousands of categories.
Takeaways & Limitations
Performance remains below the oracle setting with known bounding-box labels, and the method assumes nearest classification neighbors have similar detection models.
Abstract
from arXiv · showhide
A major challenge in scaling object detection is the difficulty of obtaining labeled images for large numbers of categories. Recently, deep convolutional neural networks (CNNs) have emerged as clear winners on object classification benchmarks, in part due to training with 1.2M+ labeled classification images. Unfortunately, only a small fraction of those labels are available for the detection task. It is much cheaper and easier to collect large quantities of image-level labels from search engines than it is to collect detection data and label it with precise bounding boxes. In this paper, we propose Large Scale Detection through Adaptation (LSDA), an algorithm which learns the difference between the two tasks and transfers this knowledge to classifiers for categories without bounding box annotated data, turning them into detectors. Our method has the potential to enable detection for the tens of thousands of categories that lack bounding box annotations, yet have plenty of classification data. Evaluation on the ImageNet LSVRC-2013 detection challenge demonstrates the efficacy of our approach. This algorithm enables us to produce a >7.6K detector by using available classification data from leaf nodes in the ImageNet tree. We additionally demonstrate how to modify our architecture to produce a fast detector (running at 2fps for the 7.6K detector). Models and software are available at
1 Introduction
LSDA addresses the shortage of bounding-box annotations by learning a general transformation from image classifiers to object detectors, using detection data for only a subset of categories. On ImageNet detection, it improves held-out-category performance over raw classifier weights.
- Research question: The central question is whether the classification-to-detection transformation can be learned on some categories and transferred to others.This extends the adaptation idea beyond settings requiring bounding-box data for every category.
- Motivation: Object detection requires extensive bounding-box annotation, which is impractical for tens or hundreds of thousands of categories.Existing datasets expanded coverage from 20 to 200 categories, but remain far smaller than the desired scale.
- Motivation: Image-level labels are easier to collect at scale, but they lack the supervised annotations needed for state-of-the-art detectors.ImageNet used search results and manual outlier detection to build a classification dataset spanning over 20,000 categories.
- Approach: LSDA treats classification data as a source domain and bounding-box detection data as a target domain, learning a transformation transferable to classifiers without detection labels.The transformation is learned from categories that have both classification and detection data.
- Results: +50% relative mAP improvement was obtained on ImageNet detection over using raw classifier weights on object proposal regions.The approach can adapt ImageNet-trained classifiers whether or not corresponding detection labels exist.
2 Related Work
Prior work established strong CNN features and classifier-to-detector adaptation, but existing object-detection adaptation methods generally require labeled detection data for all categories. LSDA instead targets held-out categories with no detection annotations.
- Related weak supervision: Weak-label multiple-instance learning is a complementary strategy, but the cited methods had not been evaluated on the large-scale ImageNet detection challenge for direct comparison.MIL uses image-level positive bags containing at least one positive region and estimates latent example labels.
- CNN foundations: CNN features became state of the art across classification and several transfer tasks, motivating their use as a foundation for detection.The cited examples include subcategory classification, scene classification, domain adaptation, and image matching.
- Domain adaptation: Domain adaptation methods modify models or features to address differences between training and test distributions, and LSDA frames classification-to-detection transfer in this way.Earlier adaptation approaches include feature-space, model, and joint feature-model transformations.
- Gap addressed: Existing supervised detection-adaptation methods require labeled detection data for all source and target categories, unlike LSDA’s held-out-category setting.The paper presents LSDA as the first method, to its knowledge, to adapt to categories without detection data.
3 Large Scale Detection through Adaptation (LSDA)
LSDA combines category-invariant and category-specific adaptations to transform classifiers into detectors, using bounding-box data for only a subset of categories. It applies the learned changes to categories lacking detection annotations and produces region-level detection scores.
- Overview: LSDA produces detection networks for all target categories, including categories without bounding-box annotations at training time.The method assumes classification data for all categories while bounding-box annotations may cover only a subset.
- Key insights: The method relies on three insights: background recognition matters, category-invariant feature changes transfer, and classifier-detector differences can be category specific.These insights organize the subsequent LSDA training procedure.
- Network training: The CNN is initialized with classification pretraining and then fine-tuned on labeled regions and background boxes from categories with detection annotations.The architecture follows the CNN setup used for ImageNet classification and the region-based detection pipeline.
- Category-invariant adaptation: Fine-tuning shared layers on annotated categories induces a generic transformation that improves classifiers for categories without detection data and learns a background weight vector.The transformation is described as category invariant across the shared layers.
- Category-specific adaptation: For categories lacking detection data, LSDA estimates category-specific output changes by averaging changes from nearest annotated categories in normalized classifier-parameter space.Nearest neighbors are selected using minimal Euclidean distance between ℓ2-normalized fc8 parameters.
- Inference: At test time, the network scores each region proposal for every category and background, then combines category and background scores to produce predictions.The direct score-vector design avoids retraining SVMs and bounding-box regressors, trading a small performance loss for flexibility and reduced training time.
4 Experiments
Experiments evaluate LSDA on held-out categories without detection training data, measuring adaptation components, error types, and large-scale detection performance. LSDA improves detection by transferring category-invariant information, reducing localization and background errors, and scaling to 7,604 categories with a faster implementation.
- 4.1 Experiment Setup & Implementation Details: The evaluation uses ILSVRC2013 data with detection annotations for selected categories and held-out categories for testing.The setup uses classification and detection splits, with val2 reserved for evaluation.
- 4.2 Quantitative Analysis on Held-out Categories: 50% relative mAP improvement over the classification-only network is achieved by the full LSDA model on held-out categories.The ablation compares adaptation against applying classification weights directly to object proposals.
- 4.2 Quantitative Analysis on Held-out Categories: 16.15% mAP is obtained on held-out categories with category-invariant adaptation plus output-layer adaptation, versus 10.31% without these improvements.The improvement is statistically significant at p = 0.017; a detector trained with all category boxes provides an upper bound.
- 4.3 Error Analysis on Held Out Categories: LSDA reduces localization and background-confusion errors compared with directly using the classification network for detection.The adapted network localizes whole objects rather than only the most discriminative parts.
- 4.3 Error Analysis on Held Out Categories: Confusion with similar categories remains a prominent source of LSDA false positives, such as bicycle for motorcycle and orange for lemon.These errors are classified as Oth errors because the detector localizes an incorrect object type.
- 4.4 Large Scale Detection: A 7,604-category detector combines 200 categories with bounding-box data and 7,404 categories trained only from image-level classification labels.A spatial-pyramid-pooling implementation reduces detection time from 4 seconds to half a second per image with nearly the same performance.
5 Conclusion
LSDA transforms classifiers into detectors by learning the change between classification and detection networks from corresponding data, then applying it where detection data are unavailable. It improves held-out-category performance without bounding-box data, while remaining below oracle performance.
- LSDA learns the change from a classification CNN to a detection CNN and applies that transformation to classifiers lacking detection data.
- 50% relative improvement was achieved without seeing any bounding-box annotated data, using the adaptation algorithm.
- The approach could enable detection for tens of thousands of categories by adapting classifiers for new categories.
- A large gap to oracle performance with known bounding-box labels remains, motivating future multiple instance learning for mining object patches.