Source-linked AI summary
TS2C: Tight Box Mining with Surrounding Segmentation Context for Weakly Supervised Object Detection
Yunchao Wei, Zhiqiang Shen, Bowen Cheng, Honghui Shi, Jinjun Xiong, Jiashi Feng, Thomas Huang
TL;DR
Weakly supervised object detection struggles because MIL-based candidates often cover discriminative parts rather than complete objects. TS2C uses surrounding segmentation context and purity-completeness criteria to mine tighter candidates, achieving 48.0% and 44.4% mAP on VOC 2007 and 2012.
Problem
Weakly supervised object detection has difficulty identifying tight, complete object boxes from image-level annotations because MIL often selects discriminative object parts.
Method
TS2C ranks proposals using segmentation confidence inside each box and in its surrounding context, combining purity and completeness to select high-quality candidates.
Results
48.0% and 44.4% mAP are obtained on VOC 2007 and VOC 2012, respectively, using TS2C-selected proposals for detector training.
Takeaways & Limitations
TS2C suppresses low-quality candidates, promotes tightly covering candidates, and can be integrated into end-to-end weakly supervised detection frameworks.
Takeaways & Limitations
TS2C can produce false predictions when segmentation quality is low or semantically identical objects overlap.
Abstract
from arXiv · showhide
This work provides a simple approach to discover tight object bounding boxes with only image-level supervision, called Tight box mining with Surrounding Segmentation Context (TS2C). We observe that object candidates mined through current multiple instance learning methods are usually trapped to discriminative object parts, rather than the entire object. TS2C leverages surrounding segmentation context derived from weakly-supervised segmentation to suppress such low-quality distracting candidates and boost the high-quality ones. Specifically, TS2C is developed based on two key properties of desirable bounding boxes: 1) high purity, meaning most pixels in the box are with high object response, and 2) high completeness, meaning the box covers high object response pixels comprehensively. With such novel and computable criteria, more tight candidates can be discovered for learning a better object detector. With TS2C, we obtain 48.0% and 44.4% mAP scores on VOC 2007 and 2012 benchmarks, which are the new state-of-the-arts.
1 Introduction
Weakly supervised object detection struggles to mine complete, tight boxes because MIL favors discriminative object parts. TS2C uses segmentation context to rank candidates by purity inside the box and completeness in its surroundings.
- WSOD aims to detect objects from image-level annotations, but accurately identifying tight boxes remains difficult compared with fully supervised detection.
- MIL commonly assigns high confidence to discriminative parts rather than entire target objects, limiting complete object localization.
- Existing segmentation-assisted proposal filtering is insufficient because considering only confidence inside boxes cannot reject high-response fragments of object parts.
- The criteria define purity as high-confidence pixels inside the box and completeness as few high-confidence pixels in the surrounding context.
- TS2C evaluates candidate tightness using the box interior and surrounding region, favoring high interior objectness and low surrounding objectness.It averages segmentation-map confidence in both regions and uses their difference to rank candidates.
- 48.0% and 44.4% mAP are achieved on VOC 2007 and VOC 2012, respectively, after using TS2C-selected proposals to train the detector.
2 Related Work
Prior WSOD work uses MIL-based two-step or end-to-end frameworks, while TS2C adds segmentation-derived objectness scoring to mine tighter candidates and integrate with existing WSOD systems.
- MIL formulations treat at least one proposal in an annotated image as positive for that class and organize prior WSOD methods into two-step and end-to-end approaches.
- Two-step methods extract proposal representations, use MIL to select candidates, and then learn object detectors through iterative or latent selection procedures.
- End-to-end methods combine CNNs with MIL, including classification-localization streams and refinement branches for proposal scoring.
- TS2C’s architecture generates localization maps, converts them into pseudo segmentation masks, trains a segmentation branch, and uses its confidence maps for proposal scoring.
- Unlike fully supervised contextual approaches, TS2C uses surrounding segmentation context to suppress false-positive object parts and can be embedded into WSOD frameworks.
3 The Proposed Approach
TS2C combines image classification, weakly supervised semantic segmentation, and object detection to mine tighter object candidates for OICR-based learning. It rates proposals using segmentation confidence inside the box and in its surrounding context, then retains the highest-scoring candidates for detection training.
- Overall Architecture: The framework contains image classification, semantic segmentation, and object detection branches trained with a composite objective.Classification maps generate localization cues, which are used to produce pseudo segmentation masks for training the segmentation branch.
- OICR Backbone: OICR uses multiple instance classification to obtain proposal scores and instance refinement to assign pseudo class labels for detector learning.Refinement selects top-scoring proposals and labels proposals with high spatial overlap, while weighted losses address initialization.
- TS2C Objectness Rating: TS2C addresses OICR’s dependence on initial candidate quality by rating proposal objectness from segmentation confidence maps.The method evaluates both the region inside a proposal and the surrounding context between the proposal and an enlarged box.
- TS2C Objectness Rating: The conditional average strategy uses high-confidence surrounding pixels to prevent unrelated background pixels from lowering the surrounding objectness score.This helps suppress partial or distracting candidates whose surrounding context does not support a complete object.
- Candidate Selection and Testing: TS2C ranks candidates by O(x_i) and selects the top 200 proposals for OICR training, while classification and segmentation branches are ignored at test time.The final detections use classification outputs from the instance refinement module.
4.1 Datasets and Evaluation Metrics
Experiments use Pascal VOC 2007 and 2012, standard benchmarks for weakly supervised object detection, with separate trainval and test splits.
- Datasets: For VOC 2007, the model trains on 5,011 trainval images and evaluates on 4,096 test images.The study also performs extensive ablation analysis on VOC 2007.
- Datasets: For VOC 2012, the model trains on 11,540 trainval images and evaluates on 10,991 test images.Evaluation is conducted by submitting the test results.
4.2 Implementation Details
The implementation uses Selective Search proposals, a VGG16 ImageNet-pretrained backbone, and a Deeplab-CRF-LargeFOV-initialized segmentation branch.
- Model Configuration: Selective Search generates object proposals, while VGG16 pretrained on ImageNet serves as the backbone.Newly added layers are randomly initialized with a Gaussian distribution having µ = 0 and δ = 0.01.
- Model Configuration: The segmentation branch initializes its corresponding layers from Deeplab-CRF-LargeFOV.The implementation uses a mini-batch size of 2 images and specifies separate learning rates across training stages.
4.3 Comparison with Other State-of-the-arts
TS2C outperforms competing weakly supervised object detection approaches on PASCAL VOC, including its OICR backbone, and further improves detection after training FRCNN with selected proposals.
- TS2C outperforms OICR by 3.1% in detection AP on VOC 2007.
- TS2C surpasses the baseline Tang et al. by 2.1% and the state-of-the-art Jie et al. by 1.7% on VOC 2012.
- 48.0% and 44.4% AP are achieved on VOC 2007 and 2012, respectively, after training FRCNN detectors with TS2C proposals.The proposals are used as pseudo ground-truth bounding boxes.
- TS2C achieves 61.0% and 64.4% CorLoc on VOC 2007 and 2012, respectively.
- Visualized detections show that TS2C reduces false positives, including partial objects, compared with OICR predictions.
4.4 Ablation Experiments
Ablation experiments show that combining purity with completeness improves candidate recall and detection performance, while surrounding-context scale and overlapping same-class objects remain practical considerations.
- Purity and Completeness: Purity and completeness are proposed as segmentation-based criteria for measuring object-candidate tightness.
- Purity and Completeness: PI − PS significantly outperforms PI in recall across different candidate counts, demonstrating the effectiveness of completeness for reducing noisy candidates.In the top-1 setting, the recall upper bound is 57.9% because images may contain multiple instances.
- Purity and Completeness: 42.2% mAP is obtained with purity-only PI, while jointly using purity and completeness with PI − PS surpasses the baseline by 2.1%.
- Influence of Enlarged Scale: All tested surrounding-context ratios from 1.1 to 1.4 outperform the baseline by more than 1.4%, with the best result at ratio 1.2.
- Influence of Enlarged Scale: The completeness score can be influenced by adjacent instances when larger surrounding-context ratios are used.
- Discussion: Failure cases arise from low-quality segmentation, semantically identical instances linked together, or overlapping objects causing false predictions.
5 Conclusion and Future Work
TS2C mines tight object boxes by using surrounding segmentation context to suppress low-quality candidates and promote candidates that tightly cover target objects. Its purity and completeness criteria can filter noisy candidates and integrate into end-to-end weakly supervised frameworks, while future work targets improved mining metrics.
- TS2C suppresses low-quality object candidates and promotes high-quality candidates that tightly cover target objects.
- Purity and completeness evaluate candidate objectness from segmentation confidence to support tight-box mining.
- TS2C can filter thousands of noisy candidates and be embedded into end-to-end weakly supervised frameworks.
- Future work will design more effective metrics for mining tight boxes.