Source-linked AI summary
PCL: Proposal Cluster Learning for Weakly Supervised Object Detection
Peng Tang, Xinggang Wang, Song Bai, Wei Shen, Xiang Bai, Wenyu Liu, Alan Yuille
TL;DR
Weakly supervised object detection must localize objects using only image-level annotations, while conventional MIL can focus on object parts. The paper introduces proposal-cluster-based iterative classifier refinement in an online multi-stream network, achieving substantial improvements across evaluated benchmarks, including 48.8% mAP and 66.6% CorLoc on VOC 2007.
Problem
Weakly supervised object detection uses easier-to-obtain image-level annotations, but conventional MIL can yield part-localized proposals that do not sufficiently overlap groundtruth boxes.
Method
PCL generates spatial proposal clusters and iteratively refines instance classifiers through multiple online network streams, including cluster-as-bag supervision.
Results
PCL shows substantial and consistent improvements across PASCAL VOC, ImageNet detection, and MS-COCO, achieving 48.8% mAP and 66.6% CorLoc on VOC 2007.
Takeaways & Limitations
Proposal clusters provide the basis for refined instance classifiers that encourage discovery of whole objects rather than only discriminative parts.
Takeaways & Limitations
The most common failures involve deformation of non-rigid objects, whose parts can remain easier to detect than the complete object.
Abstract
from arXiv · showhide
Weakly Supervised Object Detection (WSOD), using only image-level annotations to train object detectors, is of growing importance in object recognition. In this paper, we propose a novel deep network for WSOD. Unlike previous networks that transfer the object detection problem to an image classification problem using Multiple Instance Learning (MIL), our strategy generates proposal clusters to learn refined instance classifiers by an iterative process. The proposals in the same cluster are spatially adjacent and associated with the same object. This prevents the network from concentrating too much on parts of objects instead of whole objects. We first show that instances can be assigned object or background labels directly based on proposal clusters for instance classifier refinement, and then show that treating each cluster as a small new bag yields fewer ambiguities than the directly assigning label method. The iterative instance classifier refinement is implemented online using multiple streams in convolutional neural networks, where the first is an MIL network and the others are for instance classifier refinement supervised by the preceding one. Experiments are conducted on the PASCAL VOC, ImageNet detection, and MS-COCO benchmarks for WSOD. Results show that our method outperforms the previous state of the art significantly.
1 INTRODUCTION
The paper addresses weakly supervised object detection by replacing conventional MIL's image-level focus with proposal clusters that support iterative instance-classifier refinement. Its online multi-stream network improves localization and achieves strong benchmark results.
- Motivation: Image-level annotations are easier to obtain than detailed bounding-box annotations, motivating weakly supervised object detection.Internet image searches can provide large collections with image-level tags.
- Problem: Conventional MIL treats images as bags and proposals as instances, but top-scoring proposals may localize object parts rather than whole objects.Detection additionally requires sufficient overlap with groundtruth bounding boxes, unlike image classification.
- Proposal clusters: Proposal clusters group spatially adjacent proposals associated with individual objects, while a separate cluster represents background proposals.Because groundtruth boxes are unavailable, cluster centers are selected from high-scoring proposals during training.
- Classifier refinement: PCL refines instance classifiers using direct object-label assignment and, more flexibly, by treating each proposal cluster as a small new bag.The small-bag strategy reduces ambiguities because not every proposal in a cluster should receive a high classification score.
- Online training: The online architecture uses a basic MIL stream followed by iterative refinement streams that generate cluster-based supervision for subsequent losses.Streams share proposal features, and refinement progressively encourages the detector to discover whole objects rather than parts.
- Results: 48.8% mAP and 66.6% CorLoc on VOC 2007 represent more than 5% absolute improvement over previous best-performing methods.Experiments also cover ImageNet detection and MS-COCO, while the paper extends earlier work with additional analyses and methodological improvements.
2 RELATED WORK
WSOD research commonly formulates detection as multiple instance learning, while this paper develops proposal-cluster-based refinement to address limitations of that formulation. The approach is trained online and end-to-end rather than through separate stages.
- Multiple Instance Learning: MIL treats images as bags and proposals as instances, learning instance classifiers from image-level bag labels.
- Weakly Supervised Object Detection: The paper learns refined instance classifiers from proposal clusters formed using instance scores and spatial relations in an online manner.
- Weakly Supervised Object Detection: Earlier WSOD methods often use CNNs as proposal feature extractors, improving performance over traditional hand-designed features.
- Weakly Supervised Object Detection: Unlike prior graph-based approaches trained step-by-step, this method generates clusters each iteration and trains its network end-to-end.
3 METHOD
The method feeds an image and roughly 2,000 proposals through shared convolutional features, then uses multiple streams for MIL classification and iterative classifier refinement. Each refinement stream receives supervision generated from the preceding stream's proposal scores and clusters.
- 3 METHOD: About 2,000 object proposals are processed through convolutional and SPP layers to produce fixed-size per-proposal feature maps.
- 3 METHOD: Proposal features branch into one basic MIL stream and additional streams that iteratively refine instance classifiers.
- 3 METHOD: Each stream generates proposal scores and clusters, which provide supervisions for the next classifier stream.
- 3 METHOD: Back-propagation optimizes the network losses to train proposal features and classifiers jointly.
3.1 Notations
The notation defines proposal boxes, classifier streams, object classes, score matrices, image labels, losses, and proposal clusters. Refined classifiers include an explicit background class, while clusters carry labels and confidence scores.
- 3.1 Notations: An image contains R proposal boxes B, and the method refines instance classifiers K times across K + 1 streams.
- 3.1 Notations: The basic classifier predicts scores in R^C×R, while each refined classifier predicts scores in R^(C+1)×R, including one background class.
- 3.1 Notations: The image label vector y records whether each of C object classes is present in the image.
- 3.1 Notations: H_k denotes supervision for the k-th instance classifier, and L_k denotes its loss function.
- 3.1 Notations: Each object cluster has a class label and confidence score, while the background cluster uses class index C + 1 and proposal-level background confidence.
3.2 Basic MIL network
The basic MIL network produces proposal scores from classification and detection branches, aggregates them into image-level scores, and trains against image labels. These scores then support proposal-cluster generation for later refinement streams.
- 3.2 Basic MIL network: The basic MIL network is compatible with any end-to-end trainable MIL method and is used to generate proposal scores and clusters for refinement.
- 3.2 Basic MIL network: Two fully connected branches produce classification and detection matrices, which are normalized by softmax operations along different directions.
- 3.2 Basic MIL network: Proposal scores are the element-wise product of classification probabilities and detection weights.
- 3.2 Basic MIL network: Image-level class scores are obtained by summing proposal scores, implementing weighted-sum pooling within the range (0, 1).
- 3.2 Basic MIL network: Training takes an image, proposal boxes, image labels, and refinement times as input, then computes losses for the basic and refined classifiers.
3.3 The overall training strategy
The network performs iterative online instance-classifier refinement through multiple output streams. Each refined classifier receives supervision generated from the preceding classifier's proposal scores and image labels during the forward pass.
- Multiple output streams implement iterative instance-classifier refinement, with each stream corresponding to a refined classifier.The architecture integrates the basic MIL network and refinement modules end-to-end.
- Supervision for the k-th refined classifier is generated from the preceding proposal scores ϕk−1 and image label y.These supervisions are denoted Hk(ϕk−1, y).
- The network optimizes the loss with respect to proposal features F and classifier parameters Wk, but not the generated supervision Hk(ϕk−1, y).The supervision is computed only during the forward process, with no gradients propagated through it.
- During each SGD iteration, proposal scores generate the current supervision in the forward pass, after which back-propagation updates F and Wk.The procedure is summarized in Algorithm 1 and uses online rather than alternating training.
3.4 Proposal cluster learning
Proposal Cluster Learning generates spatially organized proposal clusters from predicted scores and uses them to supervise refined instance classifiers. It supports direct proposal labels and cluster-as-bag labels, with iterative refinement encouraging coverage of larger object parts.
- Proposal cluster learning: Proposal cluster learning finds centers, groups proposals into object or background clusters, and generates supervision for refined instance classifiers.These three steps use proposal scores ϕk−1 and image labels y at each refinement.
- Learning refined instance classifiers: Refinement supervision can assign object labels directly to proposals or treat each proposal cluster as a small bag with a cluster-level label.The second strategy tolerates low-scoring proposals within a cluster and reduces ambiguity compared with direct labeling.
- Finding proposal cluster centers: The highest-scoring proposal method selects a top-scoring proposal for each positive class as a cluster center, while resolving proposals selected for multiple classes by retaining the highest predicted score.The score estimates whether a proposal covers at least part of an object of the class.
- Finding proposal cluster centers: The graph-based center method addresses multiple same-category objects by connecting top-ranking proposals with sufficient spatial overlap and greedily selecting well-connected centers.The number of centers can vary by image and training iteration.
- Finding proposal cluster centers: Graph edges connect proposals when their intersection-over-union exceeds the spatial-overlap threshold, and centers are greedily chosen from proposals with the most connections.The cited example threshold is It = 0.4.
- Learning refined instance classifiers: As refinement proceeds, the detector gradually detects larger object parts by forcing the network to see larger regions.The process uses multiple refinement times indexed by increasing k.
3.5 Testing
At test time, the method averages proposal scores from all refined instance classifiers and applies non-maximum suppression to remove redundant detections.
- The mean proposal-score output of all refined classifiers supplies the final detection scores during testing.The refined classifier outputs are combined rather than selecting a single stream.
- Non-maximum suppression filters redundant detections after the refined classifier scores are combined.
4 EXPERIMENTS
The experiments evaluate PCL across PASCAL VOC, ImageNet detection, and MS-COCO using image-level annotations, with VOC 2007 analyses varying refinement times and training strategies. The reported VOC analysis examines cluster-generation and refinement choices through the Fig. 5 curve families.
- Datasets and evaluation: Experiments evaluate the method on PASCAL VOC 2007 and 2012, ImageNet detection, and MS-COCO using only image-level annotations for training.
- Datasets and evaluation: PASCAL VOC evaluation reports mAP and CorLoc, while ImageNet detection uses mAP.
- Datasets and evaluation: MS-COCO evaluation uses mAP@0.5 and mAP@[.5, .95], corresponding to IoU>0.5 and averaged IoU thresholds from 0.5 to 0.95.
- Implementation: The implementation uses pretrained VGG M and VGG16 networks modified with an SPP layer and dilated convolutional layers.
- Implementation: One SGD mini-batch uses 2, 32, and 4 images for PASCAL VOC, ImageNet, and MS-COCO, respectively.
- VOC 2007 analysis: Fig. 5 compares VOC 2007 results across refinement times, highest-scoring versus graph-based clustering, direct labels versus cluster-as-bags, and online versus alternating training.The legend identifies PCL-xx-H/G, PCL-OL/OB-x, and PCL-AB-x for these settings.
4.2 Discussions
Experiments examine refinement, clustering, loss weighting, thresholds, scale, and cluster-center settings on VOC 2007. Refinement and weighted loss help substantially, while graph-based clustering and multi-scale training perform best among tested settings.
- Refinement strategy: Single refinement substantially improves performance over basic MIL, with further gains from multiple refinements before saturation after three iterations.There are no obvious improvements from three to four refinement iterations.
- Proposal clustering: Graph-based cluster-center generation outperforms highest-scoring-proposal selection across refinement counts.The method therefore uses graph-based clustering in subsequent experiments.
- Refinement strategy: Treating clusters as bags outperforms direct label assignment, while online training consistently and significantly beats alternating training.Online training also reduces extra computation by sharing proposal features.
- Loss weighting: Weighted loss achieves mAP 40.8% and CorLoc 59.6%, compared with mAP 33.6% and CorLoc 51.2% for unweighted loss.The unweighted loss provides scant improvement from refinement and performs worse than alternating training.
- Parameter settings: Setting It to 0.4 gives the best performance, while three k-means clusters slightly outperform two or four clusters.Two clusters yield mAP 40.2% and CorLoc 59.3%; four yield mAP 40.7% and CorLoc 59.6%.
4.3 Comparison with other methods
PCL-OB-G outperforms compared methods on VOC, ImageNet, and MS-COCO, including methods using more elaborate basic MIL components. Its results can improve further through model ensembles and Fast R-CNN refinement.
- VOC comparisons: PCL-OB-G outperforms other methods on VOC 2007 and 2012, even surpassing some ensembles while using a single model.The comparison includes methods with objectness, spatial regularization, or context enhancements.
- Model combinations: Ensembling VGG M and VGG16 produces little improvement, while training Fast R-CNN on ensemble top-scoring proposals further improves performance.The resulting variants are reported as PCL-OB-G-Ens. and PCL-OB-G-Ens.+FRCNN.
- ImageNet: On ImageNet detection, PCL outperforms other methods by a large margin.
- MS-COCO: On MS-COCO, PCL obtains better performance than the recent work.The authors suggest replacing that work’s conference-version component with the newly proposed method as future exploration.
4.4 Qualitative results
Qualitative analyses show that PCL generates adaptive proposal clusters and often produces more accurate detections than WSDDN variants. Remaining errors involve overlarge boxes and part detections, especially for deformable objects.
- Proposal clusters: Cluster centers contain at least parts of objects and cover an adaptive number of objects for each class.
- Comparison: PCL often provides more accurate detections than WSDDN and WSDDN+context, which sometimes produce overlarge boxes or object parts.
- Failure cases: All three methods can include adjacent similar objects or detect only parts of deformable objects.The comparison identifies these as common weakly supervised detection problems.
- Failure cases: For rigid objects, failures mainly arise from overlarge boxes; for non-rigid objects, the detector often selects less-deformed parts such as heads.The authors state that an ideal solution remains open.
- Runtime: PCL testing takes almost the same time as the basic MIL network because output streams share proposal feature computations.Extra computation mainly occurs during training for cluster-center and proposal-cluster generation.
5 CONCLUSION
The paper proposes proposal-cluster-based instance-classifier refinement for WSOD, implemented through multiple online-trained output streams. Experiments report substantial and consistent improvements, while deformation of non-rigid objects remains the main failure mode.
- Conclusion: PCL generates proposal clusters to learn refined instance classifiers for weakly supervised object detection.
- Conclusion: Multiple output streams and online end-to-end training implement iterative classifier refinement for effectiveness and efficiency.
- Conclusion: Experiments show substantial and consistent improvements, with non-rigid-object deformation remaining the most common failure case.The authors identify weakly supervised semantic segmentation as a possible future application.