Source-linked AI summary
Exploit Bounding Box Annotations for Multi-label Object Recognition
Hao Yang, Joey Tianyi Zhou, Yu Zhang, Bin-Bin Gao, Jianxin Wu, Jianfei Cai
TL;DR
Global CNN representations are not optimal for images containing multiple objects with varying categories, scales, and locations. The paper extracts object proposals, treats each image as a bag of instances, and combines standard CNN features with bounding-box-derived nearest-neighbor label information. The resulting multi-view multi-instance framework achieves state-of-the-art results on two multi-label benchmarks and can improve unseen categories using partial strong labels.
Problem
Global CNN representations are limited for multi-label images because multiple objects differ in categories, scales, locations, occlusions, and appearance.
Method
The framework treats object proposals as instances in image bags and combines a standard CNN feature view with an LMNN-based label view derived from bounding-box annotations.
Results
The framework achieves state-of-the-art results on two multi-label benchmark datasets and improves unseen categories when only partial strong labels are available.
Takeaways & Limitations
Bounding-box annotations can provide a complementary local label view that strengthens multi-label recognition and generalizes beyond categories with strong labels.
Takeaways & Limitations
The framework depends on proposal coverage for the MIL assumption and may benefit from filtering noisy proposals or removing the need for strong labels.
Abstract
from arXiv · showhide
Convolutional neural networks (CNNs) have shown great performance as general feature representations for object recognition applications. However, for multi-label images that contain multiple objects from different categories, scales and locations, global CNN features are not optimal. In this paper, we incorporate local information to enhance the feature discriminative power. In particular, we first extract object proposals from each image. With each image treated as a bag and object proposals extracted from it treated as instances, we transform the multi-label recognition problem into a multi-class multi-instance learning problem. Then, in addition to extracting the typical CNN feature representation from each proposal, we propose to make use of ground-truth bounding box annotations (strong labels) to add another level of local information by using nearest-neighbor relationships of local regions to form a multi-view pipeline. The proposed multi-view multi-instance framework utilizes both weak and strong labels effectively, and more importantly it has the generalization ability to even boost the performance of unseen categories by partial strong labels from other categories. Our framework is extensively compared with state-of-the-art hand-crafted feature based methods and CNN based methods on two multi-label benchmark datasets. The experimental results validate the discriminative power and the generalization ability of the proposed framework. With strong labels, our framework is able to achieve state-of-the-art results in both datasets.
1. Introduction
The paper addresses the difficulty of recognizing multiple objects from global CNN representations by adding local proposal-level information and bounding-box-derived label relationships. Its multi-view framework combines weak image labels, strong bounding-box labels, and CNN features, while partial strong labels can generalize to other categories.
- Motivation: Global CNN features are less suitable for multi-label images because objects vary in location, scale, occlusion, category, and intra-class appearance.These variations make classifiers trained from global representations more difficult to optimize and potentially unreliable in sparsely sampled feature-space regions.
- Motivation: Object proposals add local information by decomposing each image into regions that may contain individual objects.This avoids directly recognizing multiple objects from the whole image and instead evaluates whether target objects occur in local regions.
- Proposed framework: Partial strong labels can support generalization across the whole local-region space, making the framework more practical when bounding boxes are unavailable for every category.The strong labels are used indirectly through LMNN to encode neighborhood relationships among labeled local regions.
- Proposed framework: The framework uses weak image labels and strong bounding-box labels as complementary views for multi-label object recognition.A standard CNN supplies the feature view, while an LMNN CNN learns local nearest-neighbor relationships for the label view.
2. Related Works
Related work spans CNN-based multi-label recognition, proposal-based local representations, multi-view and multi-instance learning, and local or metric learning. These lines of research motivate combining multiple feature sources and exploiting neighborhood structure.
- CNN-based multi-label object recognition: CNN-based recognition methods use pretrained networks for global or proposal-level representations, with weakly supervised approaches aggregating proposal scores and deeper multi-scale models improving performance.The paper positions its proposal-based CNN framework within this progression from global features to localized and multi-scale representations.
- Multi-view and multi-instance learning: The proposed framework is related to multi-view learning because it combines information from multiple feature sets through view fusion.Multi-view learning commonly combines views at the feature or classifier level for supervised learning.
- Multi-view and multi-instance learning: It is related to multi-instance learning because each image is treated as a bag containing multiple object-proposal instances.The framework transforms multi-label recognition into a multi-class multi-instance problem before combining the views.
- Local and metric learning: Local and metric learning methods use nearby labeled instances or learned feature transformations to tailor models to neighborhood structure.This local-learning perspective supports the paper’s use of nearest-neighbor relationships among proposal regions.
3. Multi-Label as Multi-Instance
The paper reformulates multi-label recognition as multi-class multi-instance learning by extracting object proposals, representing them with CNN features, and aggregating proposal information into image-level Fisher vectors. Proposal coverage supports the MIL assumption, while selective search supplies proposals without extra training data or bounding boxes.
- Multi-label as multi-instance: Each image becomes a bag of proposal instances, including positive regions containing target objects and negative regions containing background or other objects.This transforms image-level multi-label classification into multi-class multi-instance learning.
- Multi-label as multi-instance: The MIL formulation relies on extracting enough proposals to cover the objects or object categories present in each image.The paper notes that several hundred proposals from modern object-detection methods can provide reasonably good recall, supporting the positive-bag assumption.
- Proposal generation: Selective search generates proposals without requiring extra training data or ground-truth bounding boxes at the proposal-generation stage.The method uses selective search because it balances effectiveness and efficiency.
- Fisher-vector representation: Proposal features are aggregated into a Fisher vector for each image and used to train one-versus-all linear classifiers.The Fisher vector concatenates proposal statistics across the Gaussian-mixture components.
4. From Global Representation to Local Similarity
The framework moves from proposal CNN features to local similarity by learning a discriminative metric from strong labels and encoding nearest-neighbor labels as a second view. These representations are combined with weak-label CNN outputs for multi-label recognition.
- Proposal feature representation: CNN features represent each object proposal, and Fisher-vector encoding aggregates proposal features into an image-level bag representation.The proposal feature view uses the second-last fully connected CNN layer; the resulting Fisher-vector baseline uses only this view.
- Metric learning: Large-margin metric learning seeks compact same-class neighborhoods while separating different classes by a large margin.The objective pulls instances toward target nearest neighbors and penalizes small distances to differently labeled instances.
- Metric learning: The LMNN CNN replaces logistic loss with a large-margin nearest-neighbor loss to produce low-dimensional features suited to local neighborhood encoding.Its output is used to measure pairwise distances and build a candidate pool from ground-truth objects.
- Label view: The label view finds k nearest ground-truth-object neighbors for each proposal and encodes their category labels as a binary 1 × kC vector.This neighborhood label representation is concatenated with the feature view using a trade-off parameter λ.
- Label view: Indirect bounding-box supervision can generalize to unseen categories by using semantically or visually related labeled objects as local support.The paper gives cat–dog/tiger and train–car/truck as examples of this neighborhood-based support.
5. Experimental Results
Experiments on VOC 2007 and VOC 2012 evaluate the framework against hand-crafted and CNN-based methods under standard and stronger training settings. Adding label-view information improves performance, generalizes from partial bounding-box annotations, and reaches state-of-the-art results in reported comparisons.
- Datasets and baselines: The evaluation uses VOC 2007 and VOC 2012, with TRAIN and VAL for training, TEST for testing, and AP and mAP as metrics.The comparisons include hand-crafted methods, 8-layer CNNs pretrained on ILSVRC 2012, very-deep CNNs, and methods using additional training data.
- Ablation settings: The framework compares feature view alone with label views using 10 or 20 categories of ground-truth bounding boxes.FeV excludes label-view features, FeV+LV-10 uses 10 annotated categories, and FeV+LV-20 uses all 20 categories.
- VOC 2007: 2.2%: FeV outperforms HCP-1000C on VOC 2007, while FeV+LV-20 adds a further 2.5% performance gain.The reported gains support the effectiveness of the feature representation and the label view, respectively.
- VOC 2007: 1.3%: FeV+LV-10 outperforms FeV on VOC 2007 using bounding boxes from only the first 10 categories.The label view also improves unseen categories such as HORSE, PERSON, and TV through local similarity representation.
- VOC 2007: 92.0% mAP: fusing FeV+LV-20-VD with VeryDeep achieves a state-of-the-art result on VOC 2007.The fusion result suggests that the proposal-based framework and whole-image multi-scale CNN features are complementary.
- VOC 2012: On VOC 2012, FeV beats NUS-PSL by 1.8% and HCP-1000C by 2.3%, while FeV+LV-20 adds another 2% performance gain.Using only 10 categories of bounding boxes, FeV+LV-10 does not degrade much; with a 16-layer CNN, FeV+LV-20-VD performs similarly to VeryDeep.
6. Conclusion
The proposed multi-view multi-instance framework combines strong-label local information with typical CNN features for multi-label classification. Experiments validate its discriminative power and generalization ability, while future work targets scalability and performance improvements.
- The framework combines bounding-box-based strong-label local information with typical CNN feature-view information for multi-label classification.
- Experimental results validate the framework’s discriminative power and generalization ability.
- Future work proposes filtering noisy object proposals to improve the framework’s scalability and possibly its performance.