Source-linked AI summary
Weakly Supervised Object Localization and Detection: A Survey
Dingwen Zhang, Junwei Han, Gong Cheng, Ming-Hsuan Yang
TL;DR
Weakly supervised object localization and detection seeks to learn object locations from limited, image-level supervision while addressing uncertainty in instance-level annotations. This survey organizes the field’s methods, development, datasets, metrics, applications, challenges, and future directions, finding that deep learning approaches generally offer stronger performance than classic methods but remain dependent on effective weakly supervised cues.
Problem
Weakly supervised localization and detection must infer instance-level object locations from image-level annotations while handling uncertainty, noisy samples, and domain variation.
Method
The survey reviews classic models, off-the-shelf deep-feature methods, and deep weakly supervised frameworks, organizing them into eight subcategories and discussing their relationships, datasets, metrics, applications, and future directions.
Results
Deep weakly supervised methods perform well without complex initialization, multi-network training can further improve performance, and classic methods are limited by feature representation and model complexity.
Takeaways & Limitations
The field’s progress has shifted toward deep learning, while effective initialization, refinement, or weakly supervised information remains central to learning useful object models.
Takeaways & Limitations
Weakly supervised action localization additionally faces large variation in action duration and is discussed as a related but distinct application area.
Abstract
from arXiv · showhide
As an emerging and challenging problem in the computer vision community, weakly supervised object localization and detection plays an important role for developing new generation computer vision systems and has received significant attention in the past decade. As methods have been proposed, a comprehensive survey of these topics is of great importance. In this work, we review (1) classic models, (2) approaches with feature representations from off-the-shelf deep networks, (3) approaches solely based on deep learning, and (4) publicly available datasets and standard evaluation metrics that are widely used in this field. We also discuss the key challenges in this field, development history of this field, advantages/disadvantages of the methods in each category, the relationships between methods in different categories, applications of the weakly supervised object localization and detection methods, and potential future directions to further promote the development of this research field.
1 INTRODUCTION
Weakly supervised object localization and detection learns from image-level or other partial annotations to reduce costly fine-grained labeling, but must propagate uncertain supervision to instance-level locations. The survey reviews the field’s challenges, two-stage learning strategy, and method categories.
- Motivation: Weakly supervised learning uses partial information and small amounts of weakly labeled data, reducing the human labor required for fine-grained annotation.It is especially useful when obtaining complete fine-grained labels is labor intensive and time consuming.
- Task scope: The survey treats weakly supervised object localization and detection as a common task because both use image-level annotations and target bounding-box-level object localization.The survey also notes that WSOD can be performed by training fully supervised detectors on locations obtained from WSOL.
- Challenges: Weakly supervised methods must handle ordinary appearance and geometric variation alongside uncertainty caused by inconsistent image-level annotations and real object locations.A central challenge is propagating image-level supervisory signals into instance-level bounding-box training data.
- Challenges: Inaccurate instance locations arise from ambiguity over whether labels correspond to object parts, whole objects, or contextual regions, producing harmful training samples.Such inferred boxes may include local object parts or undesired context.
- Challenges: Noisy samples remain problematic because bounding boxes may include background, and visually similar categories can be confused, such as bicycles and motorcycles.Without information separating foreground from background, a learner may assign background labels to object regions.
- Challenges: Domain shifts bias learning when localized training regions lack diversity in shape, appearance, scale, or viewpoint, limiting generalization to different test conditions.The survey illustrates this with swimming swans in training and flying swans at test time.
- Learning strategy: Existing detectors commonly use initialization to propagate image-level labels into noisy instance annotations, followed by refinement that gradually mines object knowledge.The two stages must collaborate to address the field’s challenges.
- Survey scope: The survey organizes approaches into classic models, off-the-shelf deep feature representations, and deep learning frameworks, with eight subcategories and proposed future directions.It also discusses relationships among categories, open problems, datasets, metrics, applications, and research directions.
2 TAXONOMY
The survey taxonomizes weakly supervised object localization and detection methods into three main categories and eight subcategories, while relating approaches across categories and tracing their historical development.
- Taxonomy: Existing methods are categorized by classic formulations, off-the-shelf deep feature representations, and deep weakly supervised learning algorithms.The taxonomy further divides these main categories into two or three subcategories each.
- Scope: The survey mainly concerns instance localization with bounding boxes, distinguishing it from earlier methods that localized category-wise key points.This defines the survey’s scope within weakly supervised localization research.
- Taxonomy: Figure 2 presents three main categories and eight subcategories on the left, and relationships between approaches in different categories on the right.The cross-category view highlights connections among the methodological families.
- Development history: A few classic-formulation approaches appeared around 2002, research progressed slowly through 2009, and many methods emerged after 2014.In recent years, deep-learning-only approaches became the mainstream direction.
- Relationships: Methods in different categories are connected because classic formulations have been combined with deep feature representations, while deep models incorporate classic formulations and feature extraction schemes.Some deep approaches are end-to-end trainable.
3 CLASSIC MODELS
Classic weakly supervised localization and detection methods typically use a two-stage framework: initialize candidate object regions, then refine the object appearance model and locations. They rely on hand-crafted features and classic detectors, enabling efficient learning on small datasets but limiting performance.
- 3.1 Initialization: Classic approaches typically combine an initialization module with a subsequent refinement process.The reviewed detectors include DPM and SVM models with hand-crafted descriptors such as HOG, SIFT, and Lab color.
- 3.2 Refinement: Refinement progressively determines object instances by optimizing learning objectives, optimization strategies, or learning mechanisms.SVM, CRF, MIL, and iterative detector-training procedures are used to refine appearance models and object locations.
- 3.3 Initialization and Refinement: Classic methods often retain the two-stage initialization-and-refinement design in later weakly supervised approaches.Iterative methods can jointly account for initialization and refinement while dynamically updating detectors and locations.
- 3.1 Initialization: Initialization mines candidate object regions using bottom-up cues or transferred top-down appearance priors.Bottom-up cues include saliency, objectness, and class consistency; top-down cues commonly come from auxiliary data with instance-level annotations.
- 3.4 Discussion: Their advantages are small-scale data requirements and fast training and testing, whereas limited feature representation and model complexity constrain performance.The survey explicitly identifies this trade-off as the main strength and weakness of classic formulations.
4 OFF-THE-SHELF DEEP MODELS
Off-the-shelf deep-model approaches replace hand-crafted representations with pretrained or fine-tuned deep features and use semantic or spatial cues to support weakly supervised localization and detection. The survey reports substantial gains over classic models, while noting that feature learning remains an opportunity for improvement.
- 4.1 Overview: These approaches use deep representations from pretrained classification models such as AlexNet and VGG, sometimes fine-tuned on target weakly labeled images.The category includes classic formulations built on deep features and methods that fine-tune off-the-shelf DNNs during weakly supervised learning.
- 4.2 Visual Cues: Pretrained networks provide visual and semantic cues, including intermediate activations, output scores, and category-consistent regions.Examples include mask-out strategies based on classification-score degeneration and descriptor correlations in convolutional activations.
- 4.3 Fine-Tuned Deep Models: Fine-tuned approaches further adapt pretrained networks to localized object instances or weakly supervised target domains.Representative methods mine locations before applying object detectors or fine-tuning classification networks on localized instances.
- 4.4 Discussion: Compared with classic models, deep features and top-down cues learned from large-scale auxiliary data yield a significant performance gain.The survey identifies feature learning as important and describes pretrained DNNs as a simple but effective source of useful cues.
5 DEEP WEAKLY SUPERVISED LEARNING
Deep weakly supervised learning methods jointly learn feature representations and object detectors within newly designed neural networks. They range from end-to-end single-network models to multi-network systems that combine specialized region-mining and detection components.
- 5.1 Overview: These methods learn both feature representations and object detectors through newly designed deep neural networks.Unlike off-the-shelf approaches, the core learning framework is learned rather than assembled around fixed pretrained representations.
- 5.2 Single-Network Training: Single-network methods discover object regions end-to-end from images or proposals supervised by image-level classification labels.They generally avoid meticulously designed initialization procedures and use the network’s learning process to identify regions.
- 5.2 Single-Network Training: MIL-based networks aggregate proposal-level scores into image-level predictions using mechanisms such as max pooling or two-stream score regularization.The networks extract proposal features, infer proposal labels, and propagate scores to image-level semantic outputs.
- 5.3 Multi-Network Training: Multi-network methods assign different networks to region mining, MIL detection, or final object detection.Some systems train these components jointly or across multiple stages, combining function-specific networks to improve learning performance.
- 5.3 Discussion: The survey reports that multi-network training can improve performance, while effectiveness remains limited by the quality of information extracted from weak supervision.Deeper backbones can also cause dramatic accuracy deterioration and training non-convergence in some weakly supervised detection frameworks.
6 DATASETS AND EVALUATION METRICS
The survey identifies public benchmarks and metrics for evaluating weakly supervised object detection and localization. PASCAL VOC, ILSVRC, and CUB are commonly used, with evaluation protocols differing between detection and localization.
- 6.1 Datasets: PASCAL VOC 2007, 2010, and 2012 are common benchmarks for weakly supervised object detection, each covering 20 object classes.Their image counts are 9,962, 21,738, and 22,531, respectively, with trainval data used for detector training and the remainder for evaluation.
- 6.2 Detection Metrics: Detection performance on PASCAL VOC is measured by mAP, with a detection counted as successful when IoU exceeds 50%.The protocol uses mean average precision over detections satisfying the stated overlap threshold.
- 6.1 Datasets: Weakly supervised localization is commonly evaluated on PASCAL VOC, ILSVRC, and CUB using protocols tailored to image-level localization.PASCAL VOC localization uses trainval images for both weak learning and localization, while ILSVRC and CUB provide larger or fine-grained evaluation settings.
- 6.2 Localization Metrics: CorLoc evaluates whether each image’s highest-scoring class-specific box overlaps the ground-truth box by more than 50%.ILSVRC 2016 contains more than 1.2 million training images across 1,000 classes, while CUB-200-2011 contains 11,788 images across 200 categories.
- 6.1 Datasets: PASCAL datasets contain complex images with multiple instances, multiple categories, and substantial object-scale variation.The survey presents dataset examples to illustrate content bias across benchmarks.
7 APPLICATIONS
Weakly supervised object localization and detection is applied across video, art, medical, and remote-sensing analysis, especially where object-level annotations are costly or scarce. Each domain introduces distinct challenges, including temporal variation, domain-specific appearance, limited expert labels, and small target objects.
- Video Understanding: Video applications reduce annotation costs for frame-level object understanding and extend weak supervision to temporal action localization.WSAL predicts temporal boundaries in weakly labeled untrimmed videos, where action duration and scene entanglement add difficulty.
- Art Image Analysis: Art-image applications address paintings, watercolor images, and classical Greek vases through weakly supervised detection and localization.Domain adaptation, specialized object classes, and distinctive visual content are central concerns in these applications.
- Medical Imaging: Medical imaging benefits from weak supervision because target-object annotations are scarce and often require trained experts.Reported applications include tuberculosis-region localization, lesion detection, microscopic-object localization, and brain-lesion detection.
- Remote Sensing Imagery: Remote-sensing imagery poses annotation and localization challenges because images are large and target objects are usually very small.Weakly supervised detectors have been applied to airplanes, vehicles, and airports collected from different sensors.
8 FUTURE DIRECTIONS
The survey identifies future directions spanning stronger multiple-instance learning, unified multitask frameworks, robust learning, advanced model families, and prior-guided deep learning. These directions aim to address uncertainty, transfer information across tasks, and avoid trivial solutions.
- Multiple Instance Learning: Future multiple-instance learning research should address set-level, key-instance-shift, and scalability issues while incorporating more advanced MIL techniques.The survey describes a progression from classic MIL with priors or regularization to breakthrough deep MIL models.
- Multi-Task Learning: Unified weakly supervised multitask learning can combine detection, segmentation, reconstruction, and depth estimation so tasks provide information to one another.Joint object segmentation and 3D shape reconstruction, as well as localization and segmentation, are cited as examples.
- Robust Learning Theory: Robust learning strategies should reduce the influence of noisy samples by progressing from easy, confident examples to harder, ambiguous ones.Curriculum, zigzag, and self-paced learning methods are reported to improve performance over conventional learning, although robust theory remains lacking.
- Reinforcement and Adversarial Learning: Deep reinforcement learning and generative adversarial learning are proposed as advanced model families for weakly supervised object localization and detection.Reinforcement learning is motivated by sequential visual search that accumulates certainty over changing fixations.
- Prior-guided Deep MIL: Prior-guided deep MIL remains underexplored despite evidence that priors help prevent weakly supervised learning from drifting to trivial solutions.Suggested priors include saliency, objectness, shape, count, masks, human actions, interactions, and co-occurrence patterns.
9 CONCLUSIONS
The survey comprehensively reviews weakly supervised object localization and detection, covering methods, benchmarks, applications, and future research directions.
- The survey reviews existing literature on weakly supervised object localization and detection.
- It organizes the field by development history, method taxonomy, and relationships between methodological categories.
- It covers benchmark datasets and widely used evaluation metrics for the field.
- It reviews applications of existing weakly supervised localization and detection algorithms.
- It identifies future directions intended to further promote development of the research field.