Source-linked AI summary
On learning to localize objects with minimal supervision
Hyun Oh Song, Ross Girshick, Stefanie Jegelka, Julien Mairal, Zaid Harchaoui, Trevor Darrell
TL;DR
The paper tackles object localization from image-level presence or absence labels instead of costly bounding-box annotations. It combines discriminative submodular cover for discovering positive windows with smoothed latent SVM refinement, achieving a 50% relative improvement in mean average precision over the current state of the art on PASCAL VOC 2007 detection.
Problem
Object detectors must be learned without costly, exhaustive bounding-box annotations, using only image-level labels indicating object presence or absence.
Method
The method discovers positive object windows with discriminative submodular cover and refines detection models using a smoothed latent SVM formulation.
Results
50% relative improvement in mean average precision over the current state of the art is reported for weakly supervised detection on PASCAL VOC.
Takeaways & Limitations
The framework constructs positive windows and improves detectors with minimal supervision, achieving state-of-the-art performance on the standard benchmark object detection dataset.
Takeaways & Limitations
The initialization assumes correct boxes are similar across positive images and absent from negative images.
Abstract
from arXiv · showhide
Learning to localize objects with minimal supervision is an important problem in computer vision, since large fully annotated datasets are extremely costly to obtain. In this paper, we propose a new method that achieves this goal with only image-level labels of whether the objects are present or not. Our approach combines a discriminative submodular cover problem for automatically discovering a set of positive object windows with a smoothed latent SVM formulation. The latter allows us to leverage efficient quasi-Newton optimization techniques. Our experiments demonstrate that the proposed approach provides a 50% relative improvement in mean average precision over the current state-of-the-art on PASCAL VOC 2007 detection.
1. Introduction
The paper addresses object localization when exhaustive bounding-box annotation is too costly, using only binary image-level labels. It combines proposal reduction, discriminative window discovery, and smoothed latent-SVM refinement, achieving a 50% relative mAP improvement over prior state of the art.
- Exhaustive bounding-box annotation is costly and error prone, motivating detector learning from weakly labeled imagery.
- The target setting provides binary image labels indicating object presence or absence, without instance-level bounding boxes.
- The method reduces candidate locations from millions to thousands per image using selective-search window proposals.
- A discriminative submodular cover discovers positive object windows, after which a smoothed latent SVM refines the detector.
- 50% relative improvement in mean average precision over the current state of the art is achieved for weakly supervised learning on PASCAL VOC.
2. Related work
Prior work frames weakly supervised detection as multiple instance learning, while related research studies visual-element discovery and co-segmentation. The paper emphasizes initialization challenges on datasets without strong object-location biases and combines discriminative covering ideas with these broader connections.
- Weakly supervised detection commonly uses multiple instance learning with image-level labels and unknown object locations.
- Earlier MIL studies often used datasets with strong object-in-the-center biases, which obscured initialization difficulties in less biased data.
- Later work on challenging datasets such as PASCAL VOC makes initialization crucial and uses heuristics including shrinking bounding boxes.
- Unsupervised mid-level visual-element discovery uses weak labels to find elements common in positive but absent from negative images.
- Co-segmentation seeks unsupervised segmentation of objects present across multiple images, and its ideas inform the paper’s submodular cover framework.
3. Problem formulation
The paper formulates detection as binary classification over rectangular windows while training receives only image-level labels. Because MIL optimization is non-convex and initialization-sensitive, the framework first constructs positive windows and then refines detectors with latent-SVM objectives.
- Images are modeled as sets of overlapping rectangular windows, reducing detection to binary classification of image windows.
- Positive image labels indicate that at least one window contains the target, whereas negative labels indicate no target instances; window labels are unavailable.
- MIL optimization typically finds local minima of non-convex objectives, making solution quality heavily dependent on initialization.
- The framework uses discriminative set multicover for initialization, then compares standard Latent SVM with a smoothed objective optimized using unconstrained smooth methods such as L-BFGS.
4. Finding objects via submodular cover
The submodular-cover stage searches for jointly present, discriminative object windows by exploiting similarity across positive images and absence from negative images. A graph-based coverage objective balances relevance, discriminativeness, and complementary appearance modes, and greedy optimization provides an approximation guarantee.
- Finding objects via submodular cover: The initialization assumes correct boxes are similar across positive images and absent from negative images, so they form dense positive-only subgraphs.
- Finding objects via submodular cover: The covering framework formalizes shared visual structure while integrating positive-versus-negative relevance to find multiple object-appearance modes.
- Graph construction: Candidate boxes are connected in a bipartite graph to their top k nearest neighbors from positive images, making neighbors compete for discriminativeness.
- Coverage objective: The covering score counts neighboring boxes covered in each positive image, with threshold t limiting redundant gains beyond t boxes per image.
- Coverage objective: The threshold t trades complementary coverage across images against selecting highly connected, potentially less complementary boxes.
- Submodularity: The function F is nondecreasing and submodular, exhibiting diminishing marginal returns as selected boxes cover overlapping neighborhoods.
- Greedy optimization: A greedy algorithm repeatedly adds the box with maximum marginal gain while seeking a representative subset of minimum cardinality.
- Greedy optimization: The greedy solution satisfies F(Ŝ) ≥ αF(V) and |Ŝ| ≤ O(log k)|S*| relative to an optimal solution.
5. Iterative refinement with latent variables
The refinement stage reviews latent SVMs and smooths their maximization over latent configurations, enabling quasi-Newton optimization while considering multiple plausible configurations.
- 5.1. Review of latent SVM: Latent SVMs select a latent configuration, typically a bounding box, by maximizing the model score over possible configurations.For object detection, the configuration set consists of bounding boxes; in deformable part models, it contains part configurations.
- 5.1. Review of latent SVM: The latent SVM objective is nonconvex and nonsmooth, which makes direct optimization difficult.The formulation uses a hinge loss over latent configurations, while the negative-example terms are convex in the model parameters.
- 5.2. Smooth formulation of LSVM: Smoothing replaces the top-1 latent configuration in each maximization with a differentiable aggregation over configurations, improving robustness to unreliable early selections.The smoothed formulation considers the top-N configurations rather than only the best one at each maximization step.
- 5.2. Smooth formulation of LSVM: Nesterov smoothing adds a strongly convex function over the probability simplex, producing a differentiable objective whose smoothing strength is controlled by µ.The smoothed function converges to the original function as µ approaches zero, and the experiments learn µ from data.
- 5.2. Smooth formulation of LSVM: The smoothed latent SVM replaces the nonsmooth latent functions and hinge loss with smoothed counterparts, allowing efficient quasi-Newton optimization.The formulation uses feature matrices whose rows correspond to latent configurations, with Euclidean-norm smoothing enabling simplex projection for computation.
- 5.2. Smooth formulation of LSVM: For large configuration sets, sparse simplex projections and order preservation support reduced top-N computations instead of exhaustive exploration.The resulting objective and gradient evaluations exploit the sparsity induced by the smoothing parameter and the number of nonzero entries in the optimal simplex vector.
6. Experiments
The experiments evaluate Smooth LSVM and weakly supervised detection on MIL and PASCAL VOC 2007 benchmarks, using average precision for detection. The proposed initialization and refinement outperform relevant comparisons, including a 50% relative mAP improvement over prior state of the art.
- Experimental setup: The experiments compare LSVM with Smooth LSVM on a multiple instance learning dataset and evaluate detection accuracy on PASCAL VOC 2007.The MIL experiment measures bag classification performance, while the PASCAL experiment measures average precision against baselines.
- Multiple instance learning datasets: Without bias, Smooth LSVM significantly outperforms LSVM on the MIL dataset; with bias, it shows modest improvement in most cases.Both methods use the same initialization for this comparison.
- Experimental setup: The method uses R-CNN features computed from the DeCAF CNN trained only with ImageNet image-level annotations.The same window features support nearest-neighbor computation and detector learning.
- Experimental setup: The evaluation reports detection average precision on the PASCAL VOC 2007 test set and compares against five weakly supervised baseline methods.Some baselines use additional information, including bounding-box or instance-level annotations.
- Weakly-supervised object detection: 29.0% class mean average precision was obtained by the proposed initialization procedure, compared with 11.6% for Siva et al.’s mining algorithm under matched evaluation conditions.The comparison uses the same window proposals, features, L2 distance metric, and PASCAL 2007 evaluation criteria.
- Weakly-supervised object detection: The initial detector trained from constructed positive windows already performs well, and optimizing the MIL objective provides further improvement on the full PASCAL 2007 dataset.The reported full-dataset comparison includes a baseline mAP of 15.0% from Russakovsky et al., for which per-class data were unavailable.
7. Conclusion
The paper presents a framework for localizing objects from one-bit image-presence labels. It constructs positive windows for initial detectors, refines them through optimization, and achieves state-of-the-art object detection with minimal supervision.
- 7. Conclusion: The framework learns to localize objects using one-bit labels indicating object presence or absence in images.It does not require instance-level bounding-box annotations.
- 7. Conclusion: The framework constructs positive windows for training initial detection models and improves those models with a refinement optimization method.The conclusion presents both stages as components of the proposed framework.
- 7. Conclusion: The method achieves state-of-the-art object-detection performance with minimal supervision on the standard benchmark dataset.The supported conclusion concerns the standard benchmark object-detection setting.