Source-linked AI summary
Instance-aware, Context-focused, and Memory-efficient Weakly Supervised Object Detection
Zhongzheng Ren, Zhiding Yu, Xiaodong Yang, Ming-Yu Liu, Yong Jae Lee, Alexander G. Schwing, Jan Kautz
TL;DR
Weakly supervised object detection avoids costly strong annotations but faces instance ambiguity, part domination, and proposal-driven memory consumption. The paper introduces a unified instance-aware, context-focused framework combining self-training, Concrete DropBlock, and sequential back-propagation. It reports state-of-the-art results across COCO and VOC benchmarks, plus ResNet and weakly supervised video-detection benchmarks.
Problem
WSOD must detect objects from weaker supervision while resolving ambiguous instances, part-focused predictions, and high memory use from redundant proposals.
Method
The framework combines instance-aware self-training, learnable Concrete DropBlock, and sequential batch back-propagation to address these three challenges.
Results
The method achieves state-of-the-art results on COCO, VOC 2007, and VOC 2012, and provides benchmarks for ResNet models and weakly supervised video object detection.
Takeaways & Limitations
The framework extends WSOD evaluation across standard image benchmarks, deeper ResNet backbones, and weakly supervised video object detection.
Takeaways & Limitations
Predictions may still be dominated by object parts in some difficult cases.
Abstract
from arXiv · showhide
Weakly supervised learning has emerged as a compelling tool for object detection by reducing the need for strong supervision during training. However, major challenges remain: (1) differentiation of object instances can be ambiguous; (2) detectors tend to focus on discriminative parts rather than entire objects; (3) without ground truth, object proposals have to be redundant for high recalls, causing significant memory consumption. Addressing these challenges is difficult, as it often requires to eliminate uncertainties and trivial solutions. To target these issues we develop an instance-aware and context-focused unified framework. It employs an instance-aware self-training algorithm and a learnable Concrete DropBlock while devising a memory-efficient sequential batch back-propagation. Our proposed method achieves state-of-the-art results on COCO ($12.1\% ~AP$, $24.8\% ~AP_{50}$), VOC 2007 ($54.9\% ~AP$), and VOC 2012 ($52.1\% ~AP$), improving baselines by great margins. In addition, the proposed method is the first to benchmark ResNet based models and weakly supervised video object detection. Code, models, and more details will be made available at: https://github.com/NVlabs/wetectron.
1. Introduction
WSOD reduces the need for costly precise bounding-box annotations by training from weaker supervision, but remains challenged by instance ambiguity, part domination, and memory consumption. The proposed unified framework addresses these issues through instance-aware self-training, context-focused dropout, and sequential back-propagation.
- Motivation: WSOD replaces costly, time-consuming, or infeasible precise bounding-box annotations with weaker supervision such as image-level category labels.It commonly formulates detection as multiple instance learning over proposal bags with learnable latent variables selecting object-covering proposals.
- Challenges: Part domination makes detectors focus on the most discriminative object parts rather than the full object extent.
- Challenges: Instance ambiguity causes detectors to miss less salient objects and merge spatially adjacent instances of the same category.These issues include rare poses, smaller scales, and grouped objects in clustered scenes.
- Proposed framework: The framework addresses the three challenges with instance-aware pseudo-ground truth, learnable Concrete DropBlock, and sequential batch back-propagation.These modules target instance differentiation, contextual object coverage, and memory use, respectively.
- Results: The method reports state-of-the-art performance on COCO, VOC 2007, and VOC 2012, with ablations, qualitative results, and a first weakly supervised video-object-detection benchmark.
2. Related work
Related work for WSOD centers on multiple-instance learning, self-training, spatial dropout, and memory-efficient back-propagation. These directions respectively address proposal selection, instance-level supervision, localization robustness, and the memory costs of deeper models or larger batches.
- Weakly supervised object detection: WSOD methods formulate images as bags of object proposals and train classification losses to select confident positive proposals.Prior work modifies initialization, regularization, and representations within this multiple-instance-learning formulation.
- Self-training for WSOD: Self-training augments WSOD with instance-level pseudo-labels, either through offline confident predictions or iterative knowledge distillation.
- Spatial dropout: Spatial dropout regularizes feature maps by dropping spatial regions and has been applied to improve localization in supervised and weakly supervised tasks.Existing variants are described as non-parametric.
- Memory efficient back-propagation: Memory-efficient back-propagation trades computation time for memory by avoiding storage of intermediate representations and recovering them with small forward passes.The approach is motivated by the memory demands of deeper models and larger batch sizes.
3. Background
The background describes a multiple-instance WSOD model that combines region-classification and class-specific detection scores, aggregates region evidence into image-level predictions, and uses online self-training for pseudo-labeling.
- Multiple-instance formulation: Given image proposals, the model produces classification and detection logits for every category-region pair.These logits are converted into two score matrices representing category assignment and region detection for each category.
- Multiple-instance formulation: The final region score is the element-wise product of classification and detection scores, combining both factors for category-region assignment.
- Training objective: Summing region scores produces image-level evidence for each category, which is optimized with an image-level classification loss.The image-level label indicates whether the category exists in the image.
- Inference: At inference, region scores are converted into detections using standard non-maximum suppression and thresholding.
- Online self-training: Online self-training uses teacher region scores to generate pseudo category labels from top-scoring regions and highly overlapping neighbors.An additional student layer is trained for region classification using these pseudo-labels.
4. Approach
The approach unifies instance-aware self-training, context-focused Concrete DropBlock regularization, and sequential computation to address WSOD ambiguity, part domination, and memory limits.
- 4.1. Multiple instance self-training (MIST): Instance-aware self-training generates diverse, representative, and instance-associative pseudo boxes by ranking proposals, selecting an image-adaptive top-p% pool, and applying non-maximum suppression.The method incorporates spatial-diversity and instance-associative constraints into pseudo-box generation.
- 4.1. Multiple instance self-training (MIST): Student blocks jointly learn classification and bounding-box regression from pseudo boxes, assigning regression targets to regions highly overlapping selected pseudo boxes.Regression is introduced to improve learning from noisy pseudo-labels and proposals.
- 4.2. Concrete DropBlock: Concrete DropBlock learns spatial masks end-to-end to drop discriminative ROI regions, encouraging detectors to use context instead of object parts.A residual block produces probability maps, which are converted into hard masks through differentiable spatial Gumbel-Softmax sampling.
- Experiments: The framework is evaluated against single-model VGG16 and ResNet results on COCO, alongside broader WSOD benchmarks and ablations.The supplied table captions identify COCO evaluations for VGG16 and ResNet models.
- 4.3. Sequential batch back-propagation: The method uses sequential batch forward and backward computation to avoid storing memory-intensive ROI feature maps and their gradients inside the Neck.ROI features are split into sub-batches, with gradients accumulated across sequential passes.
5. Experiments
Experiments evaluate the framework on COCO, VOC, and video WSOD using quantitative, qualitative, ablation, robustness, and memory analyses. Results show improved detection, recall, instance separation, context coverage, and memory efficiency, while difficult cases can remain part-dominated.
- Qualitative results: The model detects previously ignored instances, separates adjacent same-category objects, and focuses on fuller object extents in qualitative comparisons.These comparisons are made against Tang et al. [46].
- Failure cases: Some difficult cases remain dominated by object parts despite improved scores for larger boxes.The limitation is noted in the horse example.
- Module analysis: MIST significantly improves results and increases all reported Average Recall metrics, while Concrete DropBlock further improves performance over several dropout variants.Concrete DropBlock also produces the largest gains for several animal classes, including person.
- Memory efficiency: The sequential batch back-propagation approach enables training with more proposals than vanilla back-propagation and reduces memory pressure in ResNet-101 experiments.The study varies proposals from 1k to 5k and compares average iteration time and memory consumption.
- Robustness: The best MIST setting is p = 0.15 and IoU = 0.2, with final results showing small sensitivity to p and slightly greater sensitivity to IoU.These settings are used for other models and datasets.
- Video WSOD: Concrete DropBlock and MIST improvements generalize to weakly supervised video object detection on ImageNet VID.The experiments use uniformly sampled key-frames and frame-level category labels during training.
6. Conclusion
The paper addresses three major WSOD issues with corresponding solutions and reports state-of-the-art results across popular datasets. It also provides the first benchmarks for ResNet backbones and weakly supervised video object detection.
- The framework addresses three major issues in weakly supervised object detection through separate proposed solutions.
- The method achieves state-of-the-art results on COCO, VOC 2007, and VOC 2012.
- The paper is the first to benchmark ResNet backbones and weakly supervised video object detection.
Change Log
The change log records successive revisions from the initial arXiv preprint through implementation updates, figure correction, appendix reorganization, and a typo fix.
- Version 1 is identified as the arXiv preprint.
- Version 2 adds implementation details, fixes a minor mistake in Figure 8, and reorganizes the appendix for readability.
- Version 3 fixes a typo in Algorithm 1.
- Appendix scope: The appendix provides additional COCO results, VOC detection and CorLoc results, qualitative results, proposal statistics, proposal ablations, implementation details, and a video demo.
- Implementation details: Implementation sections describe VGG-16, ResNet-50/101, Concrete DropBlock, and stacked student-block configurations.
A.4. Training
Training uses SGD with dataset-specific batch sizes, learning rates, schedules, and multi-scale inputs with horizontal-flip score averaging at test time.
- VOC 2007 uses batch size 8 and initial learning rate 0.01, while VOC 2012 uses batch size 16 and initial learning rate 0.02.
- Both VOC datasets are trained for 30k iterations with learning-rate decay at 20k and 26k steps.
- COCO training lasts 130k iterations with learning-rate decay at 90k and 120k steps.
- Multi-scale inputs from 480 to 1200 are used during training and testing, with test scores averaged across scales and horizontal flips.
B. Additional quantitative results on COCO
COCO results are reported across IoU thresholds, object sizes, and detection budgets, showing that object size substantially influences detection accuracy.
- COCO evaluation reports AP, AP50, AP75, size-specific AP, and AR across multiple detection budgets and object sizes.Metrics include AP over IoU thresholds .50:.05:.95, AP50, AP75, APs, APm, APl, AR1, AR10, AR100, ARs, ARm, and ARl.
- Object size significantly influences detection accuracy, with better detector performance on large objects than smaller ones.
C.1. Per-class detection results
Per-class detection and localization results show broad strengths across VOC categories, while difficult classes remain challenging and may require strong supervision.
- Per-class detection results: Our method outperforms other WSOD methods on most categories, winning on 10 VOC 2007 classes and 14 VOC 2012 classes.
- Per-class detection results: The method is competitive with supervised models on several classes, usually within a 10% AP performance gap.Competitive classes include airplane, bicycle, bus, car, cow, motorbike, sheep, and tv-monitor.
- Per-class detection results: All WSOD methods perform significantly worse than supervised methods on boat, chair, dining table, and person.The paper attributes this difficulty to large intra-class variation and inconsistent class patterns.
- Per-class correct localization results: VOC CorLoc is computed on the training set, so it indicates over-fitting behavior rather than the true detection performance.
- Per-class correct localization results: Our method achieves the third-best VOC 2007 CorLoc result and second-best VOC 2012 result, winning 2 and 19 categories respectively.
D.1. Results on static-image datasets
Static-image experiments show improved handling of instance ambiguity and part domination, broad qualitative coverage, and the need to retain redundant proposals for accuracy.
- D.1. Results on static-image datasets: The model detects missing instances, separates grouped instances with tight boxes, and alleviates part domination relative to a baseline without the proposed modules.Examples include previously ignored monitors, sheep, cars, and persons; separated buses, motorcycles, boats, and cars; and improved whole-object localization.
- D.1. Results on static-image datasets: On COCO validation images, the model detects multiple same-category instances and different object classes while covering whole objects rather than only discriminative parts.
- D.1. Results on static-image datasets: Selective-Search and MCG generate around 2K proposals on average and occasionally exceed 5K, while sequential batch back-propagation handles these cases with ResNet-101.Other methods quickly run out of memory in these high-proposal cases.
- D.1. Results on static-image datasets: Using 95% of proposals causes a 2.8% AP drop, indicating that all proposals should be retained for best performance.
G. Additional details on video experiments
Video experiments use frame-level labels and sampled key frames, with optical-flow feature warping extending the single-frame model to exploit short-term motion.
- Video experiment setup: ImageNet VID training uses frame-level category labels and 15 uniformly sampled key frames per video, with evaluation on the standard validation set.
- Video model variants: The video system compares single-frame models with and without Concrete DropBlock and a flow-enhanced model using sequential batch back-propagation.
- Optical-flow integration: For flow-enhanced training, FlowNet2 warps neighboring-frame features into the reference frame, sums them, and feeds the result to the proposed task head.
- Qualitative video results: Additional ImageNet VID visualizations arrange frames from the same video in rows and show the model handling different object poses, scales, and viewpoints.