Source-linked AI summary
Multiple Instance Detection Network with Online Instance Classifier Refinement
Peng Tang, Xinggang Wang, Xiang Bai, Wenyu Liu
TL;DR
Weakly supervised object detection avoids labor-intensive bounding-box annotations but remains challenging with only image-level tags. The paper integrates multiple instance detection with online, multi-stage classifier refinement in one end-to-end network, achieving 47.0% mAP on VOC 2007 and substantial improvements over prior methods.
Problem
Weakly supervised object detection seeks effective detection using easier-to-collect image-level tags instead of labor-intensive bounding-box annotations.
Method
The method integrates a multiple instance detection network with multi-stage instance classifiers that propagate spatially overlapped proposal labels online and train end-to-end.
Results
47.0% mAP and 64.3% CorLoc on VOC 2007 outperformed previous best-performing methods by a large margin.
Takeaways & Limitations
The framework produced substantial and consistent improvements and may apply to other weakly supervised visual learning tasks.
Takeaways & Limitations
The method remains prone to overlarge boxes around adjacent similar objects and to detecting discriminative parts of highly deformable objects.
Abstract
from arXiv · showhide
Of late, weakly supervised object detection is with great importance in object recognition. Based on deep learning, weakly supervised detectors have achieved many promising results. However, compared with fully supervised detection, it is more challenging to train deep network based detectors in a weakly supervised manner. Here we formulate weakly supervised detection as a Multiple Instance Learning (MIL) problem, where instance classifiers (object detectors) are put into the network as hidden nodes. We propose a novel online instance classifier refinement algorithm to integrate MIL and the instance classifier refinement procedure into a single deep network, and train the network end-to-end with only image-level supervision, i.e., without object location information. More precisely, instance labels inferred from weak supervision are propagated to their spatially overlapped instances to refine instance classifier online. The iterative instance classifier refinement procedure is implemented using multiple streams in deep network, where each stream supervises its latter stream. Weakly supervised object detection experiments are carried out on the challenging PASCAL VOC 2007 and 2012 benchmarks. We obtain 47% mAP on VOC 2007 that significantly outperforms the previous state-of-the-art.
1. Introduction
The paper addresses weakly supervised object detection using only image-level tags, where end-to-end networks may recognize object parts without accurately localizing whole objects. It proposes online, multi-stage classifier refinement that propagates labels through spatially overlapped proposals and achieves strong PASCAL VOC results.
- Motivation: Weakly supervised object detection uses image tags instead of labor-intensive bounding-box annotations.Image-level annotations indicate whether an object exists in an image.
- Motivation: End-to-end weakly supervised networks can classify images while selecting small or discriminative object parts that fail the IoU>0.5 detection requirement.The top-ranking proposal may not correctly localize the object even when nearby proposals have similar scores.
- Method: Online refinement labels the top-scoring proposal and spatially overlapped proposals, propagating image-level label information to improve instance classification.Overlapped proposals may cover larger portions of an object than the initially selected part.
- Results: 47.0% mAP and 64.3% CorLoc were achieved on VOC 2007, outperforming previous best-performing methods by a large margin.Experiments were conducted on the PASCAL VOC benchmarks.
- Method: The proposed framework combines a multiple instance detection network with multi-stage instance classifiers whose later stages are supervised by preceding-stage outputs.This design iteratively enhances the discriminatory power of instance classifiers.
- Method: The OICR algorithm integrates the basic detection network and multi-stage instance classifiers into one end-to-end trainable network.Compared with alternating training, it reduces training time and boosts performance.
2. Related work
Prior WSOD methods commonly formulate detection as Multiple Instance Learning, combining image bags with proposal instances and CNN-based classifiers. Related approaches differ in how they optimize, relabel, localize, and propagate instance information.
- Multiple Instance Learning: MIL treats images as bags of proposal-based regions and learns instance classifiers using bag-level constraints.A positive bag contains at least one positive instance, while a negative bag contains only negative instances.
- Relation to prior methods: The paper differs from mi-SVM by refining labels online through spatial relations rather than alternating score-based relabelling and retraining.
- MIL optimization: Some MIL methods accelerate training with differentiable losses, while others alternate instance relabelling and classifier retraining.
- CNN-based WSOD: Existing WSOD methods combine MIL with CNNs either through off-the-shelf feature extraction or end-to-end network training.
- Localization limitations: CNN-MIL approaches using max pooling can localize objects only coarsely across object sizes and aspect ratios.
- Relation to prior methods: Its proposal labelling also differs from semi-supervised propagation because it uses spatial overlap without initially labelled instances.
3. Method
The method combines a basic multiple instance detection network with online classifier-refinement stages in one end-to-end network, using preceding-stage outputs and spatial overlap to generate supervision.
- Architecture: Proposal features are produced from convolutional feature maps through spatial pyramid pooling and two fully connected layers, then shared across multiple processing streams.The first stream trains the basic MIDN, while later streams refine instance classifiers.
- Basic multiple instance detection network: The basic MIDN branches proposal features into classification and detection streams whose element-wise product yields proposal scores and weighted image-level class scores.The two streams represent class probabilities and proposal contributions to image classification.
- Online classifier refinement: The network integrates the basic classifier and refinement stages into one end-to-end model, avoiding separately retraining classifiers and enabling shared representations.The overall loss combines the basic MIDN loss with the refinement losses.
- Online classifier refinement: OICR selects the top-scoring proposal for each present class and propagates its label to spatially overlapping proposals for supervision in the next refinement stage.Proposal overlap is measured using IoU, while non-overlapping proposals receive background labels.
- Online classifier refinement: Weighted refinement losses reduce the effect of noisy early supervision by assigning smaller weights to uncertain positive instances and larger weights to reliable ones.This weighting is intended to limit performance degradation when good positive instances are difficult to identify.
4. Experiments
Experiments on PASCAL VOC evaluate OICR, its refinement components, and comparisons with prior methods. Refinement improves detection, while weighted loss and online training provide further benefits.
- Experimental setup: Experiments use VOC 2007 and 2012 with image-level labels for weakly supervised detection evaluation.The study reports mAP on test sets and CorLoc on trainval sets.
- Classifier refinement: 35.6 mAP and 56.0 CorLoc result after one refinement, compared with 29.5 mAP and 49.9 CorLoc for the basic network.Further refinements improve results, but gains from the second to third refinement are small.
- Online refinement: OICR consistently and significantly outperforms alternative refinement while requiring one model instead of K + 1 models for K refinement rounds.The shared-representation design also reduces training time.
- Weighted loss: Weighted loss achieves much better performance than unweighted loss, whose refinement improvement is scant and worse than the alternative strategy.This supports the weighted-loss formulation used by the method.
- Error analysis: Performance is stronger for rigid classes but remains poor for cat, dog, and person, where deformation encourages detection of discriminative parts.Overlarge boxes containing adjacent similar objects are another observed failure mode for rigid objects.
5. Conclusion
The paper presents OICR as an end-to-end framework combining a basic multiple instance detection network with multi-stage instance classifiers for weakly supervised detection.
- 5. Conclusion: OICR integrates a basic multiple instance detection network and multi-stage instance classifiers into one end-to-end trainable network.The authors report substantial and consistent improvements and identify instance visual similarity as future refinement information.
1. Per-class results on VOC 2012
On VOC 2012, OICR outperforms the previous state of the art overall, with stronger results on rigid objects than on deformable ones.
- 1. Per-class results on VOC 2012: OICR outperforms the previous state-of-the-art method by a large margin on VOC 2012.It performs better on rigid classes such as bicycle, car, and motorbike, but worse on cat, dog, and person.
2. Visualization comparisons
Visual comparisons show more accurate OICR detections for several classes, while failures remain for deformable objects and context-sensitive cases.
- 2. Visualization comparisons: OICR often produces more accurate boxes than WSDDN and WSDDN+context for classes such as aeroplane, bike, and car.The comparison highlights fewer overlarge boxes or boxes containing only object parts in these examples.
- 2. Visualization comparisons: For person and other deformable objects, OICR can still detect only discriminative parts rather than the whole object.The paper suggests incorporating context information as a possible improvement.