Source-linked AI summary
Weakly Supervised Object Localization with Multi-fold Multiple Instance Learning
Ramazan Gokberk Cinbis, Jakob Verbeek, Cordelia Schmid
TL;DR
Weakly supervised object localization seeks to avoid costly bounding-box annotation by learning from image-level labels. The paper uses multi-fold MIL with high-dimensional features, contrastive foreground–background descriptors, and window refinement, achieving state-of-the-art results on VOC 2007 and similar improvements on VOC 2010.
Problem
Costly bounding-box annotations motivate learning object detectors from image-wide labels that indicate object presence without locations.
Method
The paper interleaves detector training and re-localization using multi-fold MIL, Fisher vectors, CNN features, a contrastive background descriptor, and window refinement.
Results
The method obtains state-of-the-art results on VOC 2007, with similar improvements from multi-fold MIL on VOC 2010.
Takeaways & Limitations
Multi-fold MIL effectively handles high-dimensional descriptors and supports joint use of Fisher vector and CNN features for weakly supervised detection.
Takeaways & Limitations
Weak supervision may localize the most repeatable object structure rather than the full object, as illustrated by cat heads degrading CorLoc and AP.
Abstract
from arXiv · showhide
Object category localization is a challenging problem in computer vision. Standard supervised training requires bounding box annotations of object instances. This time-consuming annotation process is sidestepped in weakly supervised learning. In this case, the supervised information is restricted to binary labels that indicate the absence/presence of object instances in the image, without their locations. We follow a multiple-instance learning approach that iteratively trains the detector and infers the object locations in the positive training images. Our main contribution is a multi-fold multiple instance learning procedure, which prevents training from prematurely locking onto erroneous object locations. This procedure is particularly important when using high-dimensional representations, such as Fisher vectors and convolutional neural network features. We also propose a window refinement method, which improves the localization accuracy by incorporating an objectness prior. We present a detailed experimental evaluation using the PASCAL VOC 2007 dataset, which verifies the effectiveness of our approach.
1 INTRODUCTION
The paper addresses object localization without costly bounding-box annotations by learning detectors from image-wide labels. It introduces multi-fold MIL, complementary descriptors, and window refinement, with evaluations on VOC 2007 and VOC 2010.
- Bounding-box annotations enable supervised object detection but are costly to acquire.
- Weakly supervised object detection uses image-wide presence or absence labels and can leverage large collections of tagged internet images.
- The method interleaves detector training and object-instance re-localization using Fisher vectors and CNN features.
- Multi-fold MIL prevents rapid convergence to poor local optima caused by high-dimensional window features, while a contrastive background descriptor distinguishes foreground from context.
- The evaluation reports state-of-the-art results on VOC 2007 and additional results on VOC 2010, which had not been used in previous work.
- Window refinement improves localization by incorporating an objectness prior, extending the method beyond its earlier version with CNN, combined-feature, and mixed-supervision experiments.
2 RELATED WORK
Related work frames weakly supervised object detection as MIL over candidate windows, with initialization and iterative learning designed to address difficult localization and local-optimum problems.
- MIL formulation: MIL treats each image as a bag of candidate windows, requiring positive images to contain at least one positive instance window.
- MIL formulation: Object detection creates vastly more examples per image than other MIL settings because possible bounding boxes grow quadratically with image pixels.
- MIL formulation: Candidate-window generation makes the large search space manageable and permits computationally expensive object models.
- Iterative learning methods: Iterative WSL typically alternates detector learning with re-localization, but selection across many images can converge to a local optimum dependent on initialization.
- Initialization methods: Initialization strategies include large image-covering windows, class-independent saliency or objectness, class-specific visual words, similarity objectives, topic models, and graph-based selection.
- Iterative learning methods: Prior iterative approaches include pairwise-similarity selection, CRF-based joint inference, standard MIL, DPM training, and background descriptors.
- Window refinement: The paper's refinement differs from earlier objectness use by updating only top-scoring final localizations with a local greedy search and an edge-driven objectness measure.
3 WEAKLY SUPERVISED OBJECT LOCALIZATION
This section describes the FV- and CNN-based representations, diagnoses why standard MIL can degenerate, and introduces multi-fold MIL and window refinement to improve weakly supervised localization.
- Representations and candidate windows: Selective search generates around 1,500 candidate windows per image, reducing the search space and filtering implausible object locations.The same candidate-window strategy is used with FV- and CNN-based representations.
- Representations and candidate windows: The detector combines window appearance with context from the image complement, using a contrastive descriptor based on background and foreground FVs.Its linear score decomposes into foreground and background contributions with opposite signs.
- Representations and evaluation: CNN features outperform FV features experimentally, while combining CNN and FV features achieves the best performance among the tested representations.The features are complementary because CNN dimensions can encode higher-level structures such as animal faces and bodies.
- Failure of standard MIL: High-capacity classifiers bias standard MIL toward re-localizing the training windows, causing degenerate localization, especially with high-dimensional descriptors.The detector used for re-localization is trained on positives from the same images, and high-dimensional FV windows can be nearly orthogonal even within images.
- Multi-fold MIL: Multi-fold MIL divides positive images into K folds and re-localizes each fold with a detector trained on the other folds before retraining on all selected windows.The final detector is then used for hard-negative mining on negative images.
- Window refinement: Window refinement uses an objectness measure from edge maps to replace initial boxes with windows containing more complete object contours.The refined windows are likely better aligned with objects, making their horizontal mirrors more reliable as additional training examples.
4 EXPERIMENTAL EVALUATION
On VOC, multi-fold MIL consistently outperforms standard MIL across feature types and metrics, addressing standard MIL’s tendency to stagnate or lock onto poor windows. Combining FV and CNN features and refining windows further improves localization and detection, while remaining errors often reflect localization of object subregions.
- Multi-fold MIL outperforms standard MIL for all five tested representations in both CorLoc and AP.
- CNN features outperform FV for 12 classes in CorLoc and 13 classes in AP under multi-fold training, while benefiting less from multi-fold training than FV.
- 34.4% CorLoc and 22.0% mAP for standard MIL rise to 47.3% CorLoc and 27.4% mAP with multi-fold MIL on combined features.
- Multi-fold MIL progressively localizes smaller regions, whereas standard MIL often remains stuck after the first re-localization; this succeeds on cluttered, nearby instances but can select cat faces.
- The cat failure illustrates a scope limitation: weak labels favor repeatable distinctive structures, so localizing a head can reduce CorLoc and AP despite detecting the object category.
- Ten MIL iterations suffice for stabilization, and 10-fold training is selected because 20 folds provide limited additional gain.
- For CNN features, multi-fold training reaches 43.8% CorLoc versus 40.3% for standard MIL; changing SVM cost cannot eliminate high-dimensional-FV degeneration.
- Window refinement significantly improves average CorLoc and AP for FV, CNN, and combined descriptors, while FV+CNN reaches 30.2% detection mAP against 22.7%, 24.6%, and 26.4% competing results.
5 CONCLUSIONS
The paper introduces multi-fold MIL, a contrastive background descriptor, and edge-driven window refinement for weakly supervised object detection. Evaluations report state-of-the-art results and quantify multi-fold MIL’s effectiveness with combined FV and CNN features.
- Multi-fold MIL avoids the degenerate localization performance observed without it.
- A contrastive background descriptor encourages the detector to learn differences between objects and their context.
- Edge-driven window refinement improves localization accuracy by using an objectness prior.
- Multi-fold MIL handles high-dimensional descriptors and enables state-of-the-art results by jointly using FV and CNN features.Similar improvements are also observed on VOC 2010.
- 68% of the MIL performance upper-bound is attained in test set detection performance for combined FV and CNN features.The upper-bound is measured by selecting one correct training example from each positive image.