Source-linked AI summary
Deep Self-Taught Learning for Weakly Supervised Object Localization
Zequn Jie, Yunchao Wei, Xiaojie Jin, Jiashi Feng, Wei Liu
TL;DR
Weakly supervised localization must identify object boxes from image-level annotations, but classification features lack spatial information and can yield poor positives. The paper uses deep self-taught learning with spatially informed seeds, online detector-guided harvesting, and relative score improvement, and reports consistent outperformance on PASCAL 2007 and 2012.
Problem
WSL lacks bounding-box annotations, while classification-trained features do not encode object spatial locations and can provide poor positive samples for detector training.
Method
The approach transfers image-level information to object proposals, selects spatially concentrated seed positives with dense subgraph discovery, and iteratively harvests supportive samples using detector scores and relative improvement.
Results
The proposed approach consistently outperforms state-of-the-art methods by an obvious margin across all evaluation scenarios on PASCAL 2007 and 2012.
Takeaways & Limitations
Deep self-taught learning provides a WSL paradigm that progressively improves the quality of positive samples during detector training.
Takeaways & Limitations
Online alternating training remains vulnerable to poor local optima when weak initial seeds cause the stronger Fast R-CNN detector to overfit.
Abstract
from arXiv · showhide
Most existing weakly supervised localization (WSL) approaches learn detectors by finding positive bounding boxes based on features learned with image-level supervision. However, those features do not contain spatial location related information and usually provide poor-quality positive samples for training a detector. To overcome this issue, we propose a deep self-taught learning approach, which makes the detector learn the object-level features reliable for acquiring tight positive samples and afterwards re-train itself based on them. Consequently, the detector progressively improves its detection ability and localizes more informative positive samples. To implement such self-taught learning, we propose a seed sample acquisition method via image-to-object transferring and dense subgraph discovery to find reliable positive samples for initializing the detector. An online supportive sample harvesting scheme is further proposed to dynamically select the most confident tight positive samples and train the detector in a mutual boosting way. To prevent the detector from being trapped in poor optima due to overfitting, we propose a new relative improvement of predicted CNN scores for guiding the self-taught learning process. Extensive experiments on PASCAL 2007 and 2012 show that our approach outperforms the state-of-the-arts, strongly validating its effectiveness.
1. Introduction
Weakly supervised localization uses only image-level annotations, making reliable object-level positives difficult to obtain. This paper addresses the problem with deep self-taught learning that progressively harvests better samples through spatially aware initialization and detector-guided re-localization.
- Motivation: WSL learns object localization from image-level annotations, without expensive bounding-box supervision.This limited supervision makes reliable positive bounding-box identification critical.
- Motivation: Classification-trained CNN features lack object spatial information and therefore provide unsuitable localization cues.Existing methods also rely on offline MIL before detector training, limiting use of the detector’s discriminating ability.
- Deep Self-Taught Learning: The proposed deep self-taught learning approach alternates detector training with online supportive-sample harvesting to progressively improve positive samples.The detector uses its own predicted scores to harvest confident, tight proposals during training.
- Seed Proposal Acquisition: Image-to-object transferring and dense subgraph discovery acquire spatially aware seed positives from high-response candidate proposals.Dense subgraph discovery incorporates spatial correlations among candidate object proposals.
- Supportive Sample Selection: Relative CNN score improvement selects supportive samples while reducing the risk of poor local optima caused by overfitting to seed positives.The strategy distinguishes score gains from model improvement from gains caused by overfitting.
2. Related Work
Prior WSL methods mainly use MIL-based optimization or end-to-end CNN models. The paper positions its approach against these methods by combining spatially informed proposal initialization with online regional-detector harvesting.
- MIL-Based Methods: Most existing WSL methods formulate localization as MIL, alternating between learning object representations and selecting positive samples.Because the optimization is nonconvex, solutions can be sensitive to initial positive samples and become trapped in local optima.
- MIL-Based Methods: Some methods improve iterative optimization using held-out sample updates, posterior regularization, or multi-fold training.These strategies target local-optimum and initialization issues in iterative WSL training.
- End-to-End CNN Models: End-to-end CNN approaches use separate classification and localization streams or contextual guidance to refine proposal scores and boundaries.These models represent an alternative to conventional MIL formulations.
- Closest Related Work: The closest prior work combines whole-image classification, mask-out proposal selection, MIL, and Fast R-CNN training.The paper contrasts this pipeline with image-to-object transferring and online supportive-sample harvesting in a regional CNN detector.
3. Deep Self-Taught Learning for WSL
The approach initializes WSL with spatially aware seed proposals, then progressively improves a regional detector by harvesting supportive samples online using relative CNN-score improvement.
- Seed proposal acquisition: Image-to-object transferring learns spatial-aware features from image-level annotations to identify high-response candidate proposals.Cross-proposal max-pooling preserves the highest class response and back-propagates classification error through the most confident proposal.
- Seed proposal acquisition: Dense subgraph discovery connects high-response proposals whose IoU exceeds a threshold and selects spatially concentrated proposals through iterative degree-based pruning.The method retains an adaptive number of proposals and avoids relying solely on predicted response, unlike clustering and NMS.
- Seed proposal acquisition: The highest-response proposal among the selected spatially concentrated proposals becomes the seed positive sample for each image.
- Online detector self-training: Online supportive sample harvesting uses a Fast R-CNN detector trained on seed proposals to progressively select confident tight positives for further training.The detector and newly selected samples improve one another during self-taught learning.
- Online detector self-training: The self-taught strategy addresses the risk that alternating detector training and re-localization becomes trapped in poor local optima from poor initial seeds.
- Online detector self-training: Relative improvement ranks proposals by CNN-score change over the previous epoch, separating detector improvement from score gains caused by overfitting to false positives.The proposal with maximal relative improvement is selected for the current training epoch.
4. Experiments
Experiments on PASCAL VOC 2007 and 2012 evaluate the method using CorLoc and AP, with ablations isolating dense subgraph discovery, online supportive sample harvesting, and relative score improvement. The approach achieves strong benchmark performance, while component studies show that these mechanisms improve localization and detection.
- Datasets and evaluation: The method is evaluated on PASCAL VOC 2007 and 2012 using CorLoc for training-set localization and AP for test-set detection.PASCAL 2007 uses trainval for training and test for evaluation; PASCAL 2012 uses train/val and trainval/test settings.
- Ablation studies: DSD improves CorLoc by nearly 4% over HCP, while OSSH1 adds 12% CorLoc improvement by harvesting tight positives after one online update.Later OSSH epochs provide smaller additional CorLoc benefits than the second-epoch update.
- Ablation studies: DSD and OSSH1 improve mAP by around 3% and 9%, respectively, while NR contributes a further 1% mAP improvement by discarding false positives.Adding DSD and OSSH also produces significant mAP improvements on the PASCAL 2012 val set.
- Score-selection comparison: Relative CNN score improvement consistently outperforms absolute CNN scores for harvesting confident positives, especially when OSSH runs for more epochs.Absolute scores allow seed positives to retain high scores after overfitting, increasing the risk of poor local optima.
- Benchmark comparisons: 56.1% CorLoc on PASCAL 2007 trainval and 41.7% mAP on PASCAL 2007 test outperform the reported state-of-the-art methods.The comparison includes MIL-based methods and an end-to-end WSL network.
- Qualitative results: Qualitative examples show localization improving from HCP through DSD and OSSH, including recovery from false-positive seeds toward tighter ground-truth objects.The figure compares HCP, HCP+DSD, and three OSSH variants; green boxes denote ground truth and red boxes detected objects.
5. Conclusions
The paper concludes that deep self-taught learning combines reliable seed acquisition with online supportive-sample harvesting for weakly supervised localization. Experiments on PASCAL 2007 and 2012 report consistent improvements over state-of-the-art methods across evaluation settings.
- Contributions: The approach acquires seed positives through image-to-object transfer and dense subgraph discovery, then improves them through online supportive sample harvesting and relative CNN score improvement.These stages progressively provide higher-quality positive samples for detector training.
- Conclusions: The proposed approach consistently outperforms state-of-the-art methods by an obvious margin across evaluation scenarios on PASCAL 2007 and 2012.The conclusion attributes the results to experiments on both benchmark datasets.