Source-linked AI summary
Soft Proposal Networks for Weakly Supervised Object Localization
Yi Zhu, Yanzhao Zhou, Qixiang Ye, Qiang Qiu, Jianbin Jiao
TL;DR
Weakly supervised localization must infer object locations from image-level labels rather than bounding boxes, while conventional proposal methods can be expensive and difficult to optimize jointly. The paper integrates a Soft Proposal module into CNNs so evolving objectness maps are projected back into features and jointly learned with network parameters. SPNs significantly improve weakly supervised localization and classification, with reported gains of 5.8% mAP on VOC2012 and 6% mAP on COCO2014 over the state of the art.
Problem
Weakly supervised localization has only image-level labels, while existing proposal-and-classification pipelines are costly or cannot be jointly optimized.
Method
Soft Proposal Networks generate objectness confidence maps from deep features, project them back, and jointly evolve proposals with CNN filters using image-level supervision.
Results
SPNs significantly outperform state-of-the-art methods on weakly supervised localization and classification, including 5.8% mAP on VOC2012 and 6% mAP on COCO2014.
Takeaways & Limitations
Coupling object proposals with network learning improves object-centric evidence and weakly supervised localization and classification performance.
Takeaways & Limitations
The proposal-generation random walk usually requires about ten iterations to reach its stable state.
Abstract
from arXiv · showhide
Weakly supervised object localization remains challenging, where only image labels instead of bounding boxes are available during training. Object proposal is an effective component in localization, but often computationally expensive and incapable of joint optimization with some of the remaining modules. In this paper, to the best of our knowledge, we for the first time integrate weakly supervised object proposal into convolutional neural networks (CNNs) in an end-to-end learning manner. We design a network component, Soft Proposal (SP), to be plugged into any standard convolutional architecture to introduce the nearly cost-free object proposal, orders of magnitude faster than state-of-the-art methods. In the SP-augmented CNNs, referred to as Soft Proposal Networks (SPNs), iteratively evolved object proposals are generated based on the deep feature maps then projected back, and further jointly optimized with network parameters, with image-level supervision only. Through the unified learning process, SPNs learn better object-centric filters, discover more discriminative visual evidence, and suppress background interference, significantly boosting both weakly supervised object localization and classification performance. We report the best results on popular benchmarks, including PASCAL VOC, MS COCO, and ImageNet.
1. Introduction
Weakly supervised localization must learn object locations from image-level labels, yet conventional proposals are costly or cannot be jointly optimized with classification. The paper introduces Soft Proposal Networks, which integrate evolving object proposals into CNNs for efficient end-to-end learning.
- Weakly supervised localization provides only image-level object-presence labels, not bounding boxes, making localization substantially constrained.
- Conventional proposal-and-classification pipelines are sub-optimal because their proposal and classification steps cannot be jointly optimized.
- Soft Proposal (SP) upgrades standard CNNs to Soft Proposal Networks by generating objectness maps, projecting them onto deep features, and jointly optimizing proposals with network parameters.
- ∼0.9ms per image makes SP 10× faster than RPN and 200× faster than EdgeBoxes while retaining nearly cost-free proposal generation.
- SP proposals are soft because they assign objectness to receptive fields, couple probabilistically with activations, avoid threshold tuning, aggregate information, and evolve with CNN filters.
- The upgraded CNN-S, VGG16, and GoogLeNet models improve the state of the art in weakly supervised object localization by a significant margin.
2. Related Work
Prior weakly supervised localization methods either rely on pipelined proposals and latent-variable learning or use unified CNNs whose discriminative filters can miss object evidence and follow background cues. SPN addresses this gap by making object proposals image-specific confidence maps that evolve jointly with convolutional learning.
- Pipelined methods: Pipelined methods first extract candidate proposals and then learn classifiers while selecting proposals, but proposal quality and classifier learning remain separated.
- Object Proposal: Conventional proposals range from hand-crafted redundant boxes in Selective Search and EdgeBoxes to deep RPN regressions whose localization capability depends on a well-trained network.
- Object Proposal: SPN represents proposals as objectness confidence maps rather than materialized boxes, coupling them with convolutional activations as deep features evolve.
- Weakly Supervised Localization: The paper claims to be first to integrate proposal generation, object-region selection, and object-detector estimation through joint updates under weak supervision.
- Unified frameworks: Unified CNN methods interpret deep feature responses as localization evidence, but their filters can miss useful object regions and be misled by complex backgrounds.
3. Soft Proposal Network
Soft Proposal Networks generate objectness maps from deep feature maps, couple them with activations, and jointly update proposals and CNN filters using image-level supervision. The resulting pipeline integrates proposal generation, feature selection, classification, and weakly supervised localization in an end-to-end architecture.
- Soft Proposal Generation: SPNs generate an objectness proposal map through graph propagation over receptive fields, then iteratively update it until reaching a stable state.The graph is fully connected, and random-walk propagation typically stabilizes in about ten iterations.
- Soft Proposal Coupling: The proposal map is coupled element-wise with each deep feature map, producing features that are passed forward for image-label prediction.The coupling uses the Hadamard product V_k = U_l_k ◦ M, while the resulting prediction error is computed from image labels.
- Soft Proposal Coupling: SPN training jointly updates convolutional filters and proposal maps because the maps depend on deep features while the features depend on those filters.Backpropagation apportions gradients through the proposal map, allowing proposal generation, coupling, and filter learning to evolve together.
- Weakly Supervised Activation: Weakly supervised activation aggregates deep feature maps into image-category predictions and computes class response maps by weighting coupled feature maps.The architecture uses spatial pooling and a fully connected layer, with response maps formed from class-specific channel weights and the proposal map.
- Soft Proposal Coupling: The proposal map highlights informative and worth-learning locations, helping SPNs learn more informative image regions and depress noisy backgrounds.The map also flows with gradients, so an SP module inserted after the top convolutional layer can affect all CNN filters.
4. Experiment
Experiments show that SPNs generate high-quality proposals with negligible overhead and improve weakly supervised pointing and bounding-box localization across challenging scenes and benchmarks.
- 4.1. Proposal Quality: The proposal maps highlight small objects and fine-grained evidence, including limbs, paws, texture fragments, and partially occluded object regions.
- 4.1. Proposal Quality: 62.2% Object Energy for weakly supervised SPN proposals approaches supervised RPN's 63.2%, while exceeding Selective Search and EdgeBoxes.
- 4.2. Pointing Localization: 87.5% versus 80.0% pointing accuracy for SP-VGGNet over c-MWP on VOC2007 All, rising to 78.1% versus 66.8% on Diff.
- 4.2. Pointing Localization: SPN's accuracy gains are larger on the difficult subset, indicating particular effectiveness in cluttered scenes.
- 4.2. Pointing Localization: 5.8% mAP on VOC2012 and 6% mAP on COCO2014 over the state-of-the-art method demonstrates accurate localization while retaining classification ability.
- 4.3. Bounding Box Localization: SPN improves mean CorLoc over ContextLoc by about 5% and gains up to 20–30% on dog, cat, horse, and person classes.
5. Conclusions
Soft Proposal integrates nearly cost-free object proposals into CNNs, creating SPNs that jointly learn proposals and network parameters for weakly supervised localization.
- SPNs significantly outperform state-of-the-art methods on weakly supervised localization and classification tasks.The unified procedure generates proposals from deep feature maps, projects them back, and helps filters discover more fine-detailed evidence.