Source-linked AI summary
Two-Phase Learning for Weakly Supervised Object Localization
Dahun Kim, Donghyeon Cho, Donggeun Yoo, In So Kweon
TL;DR
Weakly supervised FCNs trained with image-level labels often highlight only the most discriminative object parts, limiting object-extent localization. The paper trains a second network while suppressing regions highlighted by the first, then combines their heat maps. Experiments report effectiveness across segmentation, saliency detection, and object-location prediction, including an mIoU gain over SEC on VOC-val.
Problem
Weakly supervised FCNs using image-level annotations tend to find the most discriminative object parts rather than the complete object extent.
Source: §7 · p. 7Method
Results
Experiments support effectiveness across segmentation, saliency detection, and object-location prediction; on VOC-val, the method improves mIoU over SEC by 2.4%.
Source: §4.6 · p. 5, §7 · p. 8Takeaways & Limitations
Two-phase learning is not limited to the SEC model or CAM module and can apply to other segmentation systems.
Source: §4.6 · p. 6The method can confuse almost-always co-occurring objects, and newly found regions rarely may belong to a class other than the predicted one.
Source: §4.6 · p. 6
Abstract
from arXiv · showhide
Weakly supervised semantic segmentation and localiza- tion have a problem of focusing only on the most important parts of an image since they use only image-level annota- tions. In this paper, we solve this problem fundamentally via two-phase learning. Our networks are trained in two steps. In the first step, a conventional fully convolutional network (FCN) is trained to find the most discriminative parts of an image. In the second step, the activations on the most salient parts are suppressed by inference conditional feedback, and then the second learning is performed to find the area of the next most important parts. By combining the activations of both phases, the entire portion of the tar- get object can be captured. Our proposed training scheme is novel and can be utilized in well-designed techniques for weakly supervised semantic segmentation, salient region detection, and object location prediction. Detailed experi- ments demonstrate the effectiveness of our two-phase learn- ing in each task.
1. Introduction
Image-level supervision leaves weakly supervised FCNs prone to highlighting only the most discriminative object parts, limiting localization of full object extent. Two-phase learning suppresses first-pass regions during second-network training, then combines both networks' activations to improve object coverage across segmentation, saliency detection, and localization.
1. Introduction
- Figure 1 contrasts the first network's heat map and baseline segmentation with the proposed method's heat map and segmentation prediction.The figure also includes an input image, estimated most- and next-important locations, and ground-truth segmentation.Source: Figure 1 · p. 1
- Image-level annotations leave weakly supervised FCNs focused on the most discriminative parts rather than the complete extent of objects.The introduction frames this as a persistent limitation across weakly supervised localization, detection, and semantic segmentation.Source: §1 · p. 1
- Two-phase learning trains a conventional FCN first, then suppresses first-stage highlighted regions while training another FCN to find other discriminative parts.At inference, activations from both phases are combined to capture the entire object portion.Source: §1 · pp. 1–2
- Combined heat maps are used to improve per-class saliency detection, object localization, and semantic segmentation.The paper presents the training scheme as applicable to these three tasks.Source: §1 · p. 2
2. Related Works
Prior approaches seek broader object coverage by adjusting score aggregation or using external saliency masks and object proposals. The proposed method instead trains for more comprehensive object features without prior object-size knowledge or external modules requiring annotations beyond image labels.
2. Related Works
- Prior work on capturing whole-object extent falls into two broad approach types.The section organizes the literature around its shared goal of going beyond the most important object's location.Source: §2 · p. 2
- Pooling-based methods balance global max pooling and global average pooling, but depend on a user parameter that predetermines the image portion of interest.Examples include log-sum-exp pooling and global weighted ranking pooling.Source: §2 · p. 2
- A second group uses external saliency masks or region proposals, while one network-derived saliency approach still involves human annotation for higher performance.The cited helpers include selective search, MCG, BING, and other proposal methods.Source: §2 · p. 2
- Unlike these approaches, two-phase learning collects more comprehensive class-related regions without prior object-size knowledge or external modules requiring lower-than-image-level annotations.The paper distinguishes its focus on finding comprehensive object features from methods that modify score aggregation.Source: §2 · p. 2
3. Two-Phase Learning
The method trains a first FCN to produce class heat maps, uses thresholded first-network maps to mask activations in a second FCN, and combines both networks' heat maps at inference. The experiments use Pascal VOC 2012 and retain two phases because localization performance decreases with further phases.
3. Two-Phase Learning
- The two-phase method comprises first-phase learning, inference conditional feedback, second-phase learning, and an inference step that combines both networks' heat maps.The first and second networks refer to the networks trained in the corresponding learning phases.Source: §3 · p. 2, §3.6 · p. 4
3.1. Dataset
- Training uses 10,582 weakly annotated Pascal VOC 2012 trainaug images, rescaled to 321 × 321.The configuration follows and the image size used in [18].Source: §3.1 · p. 2
3.2. Baseline Architecture
- The baseline is an FCN based on [42], with a modified VGG variant, global average pooling, and a 20-way fully connected layer.The authors say their approach can be applied to existing FCNs beyond this architecture.Source: §3.2 · p. 2
3.3. First Phase Learning
- In the first phase, an FCN is trained with multi-label logistic loss for 20 foreground classes and produces class-specific heat maps.The first network's heat maps provide the class-region cues used by the later feedback step.Source: §3.3 · pp. 2–3
3.4. Inference Conditional Feedback
- Inference conditional feedback selects maps for classes present in the image and thresholds them at 60% of their maxima to form a binary suppression mask.Pixels above the threshold are assigned zero and other pixels one; masks for multiple present categories are combined with logical AND.Source: §3.4 · p. 3
3.5. Second Phase Learning
- During second-phase training, the fixed first network supplies a mask that is multiplied element-wise with each second-network conv5-3 channel, dropping previously highlighted activations and gradients.Without the most discriminative first-phase features, the second network learns other category-discriminative features and reveals additional regions.Source: §3.5 · pp. 3–4
- The authors consider only two phases because localization performance gradually decreases as phases proceed; the third-phase threshold is 40%.The paper notes that additional phases could use further inference conditional feedback with a lower threshold.Source: §3.5 · p. 4
3.6. Inference
- At inference, feedback is not applied; both networks produce heat maps in a single forward pass, with combination depending on the task.The paper describes separate combination implementations for semantic segmentation, saliency detection, and object localization.Source: §3.6 · p. 4
4. Semantic Segmentation Experiments
Two-phase localization cues are integrated into SEC to improve weakly supervised semantic segmentation. The section evaluates how the cues are built and merged, and reports segmentation results including an application beyond SEC.
4.1. Review of SEC Architecture
- SEC combines seed localization cues, expand aggregation through global weighted rank pooling, and constrain regularization with dense CRFs.The three principles support localization cues, image-level score aggregation, and boundary-conforming segmentation, respectively.Source: §4.1 · p. 4
4.2. Two-phase Learning for Localization Cues
- SEC localization cues are class-specific binary masks thresholded at 20% of each heat map’s maximum; two-phase heat maps provide more comprehensive cues.Conventional FCN cues are considered reliable only for object positions, while two-phase cues are described as more powerful for segmentation.Source: §4.2 · p. 4
4.3. Merging Heat Maps from Both Phases
- Weighted map voting multiplies each phase’s class heat map by its class probability, then takes the pixel-wise maximum of the resulting maps.The method treats the first network’s class probability as confidence in its corresponding heat map.Source: §4.3 · pp. 4–5
4.6. Results and Discussion
- 53.1% mIoU on VOC-val and 53.8% on VOC-test were the best among fair weakly supervised comparisons, improving on SEC by 2.4% and 2.1%, respectively.The results are reported in Table 1 for validation and Table 2 for test.Source: Table 1 · p. 5, Table 2 · p. 5, §4.6 · p. 5
4.4. Improving Segmentation Network
- The original SEC reaches 50.7% average intersection-over-union with all three losses; the improved localization cues allow training with only seed and constrain losses.The original three-loss result is 0.3% higher than SEC trained with only seed and constrain.Source: §4.4 · p. 5
4.5. Evaluation
- Segmentation performance is evaluated by intersection-over-union on 1,449 Pascal VOC 2012 validation images.The evaluation follows the Pascal VOC 2012 semantic segmentation challenge protocol.Source: §4.5 · p. 5
4.6. Results and Discussion
- The method finds more object parts and instances while avoiding unreasonable expansion, but can confuse co-occurring objects or include occasional irrelevant regions.Examples include recovering people’s torsos, arms, and legs, and confusing trains with tracks.Source: §4.6 · pp. 5–6
- On CCNN, the method reaches 35.7% mIoU on VOC-val versus the reproduced baseline’s 34.5%, supporting applicability beyond SEC.The reported improvement comes from highlighting second-most-important parts not found by the baseline.Source: §4.6 · p. 6
5. Per-Class Saliency Prediction Experiments
The section evaluates per-class heat maps as saliency maps and combines the first- and second-network maps using weighted map voting. The combined maps achieve higher AP than first-network maps alone.
5. Per-Class Saliency Prediction Experiments
- The first- and second-network heat maps are combined by weighted map voting to test whether they jointly capture complete objects.The maps are treated as per-class saliency maps and compared with ground-truth segmentation masks.Source: §5 · p. 6
5.1. Evaluation
- The evaluation collects 2,148 per-class heat-map and ground-truth saliency-mask pairs from 1,440 Pascal VOC 2012 validation images and measures average precision.Only classes present in each image are included; pixel responses are treated as confidence values for precision-recall curves.Source: §5.1 · p. 6
5.2. Results
- Weighted map voting combines a set of our heat maps to achieve AP of 37.7%, 5.5% higher than the 32.5% achieved using only the first heat maps.Source: §5.2 · p. 6
6. Location Prediction Experiments
The experiments assess whether the second network highlights distinct, object-relevant locations and whether combining phases improves object coverage. The second network retains strong localization performance while often finding different object parts from the first.
6. Location Prediction Experiments
- The second network is intended to reveal valuable features not discovered in the first phase, complementing first-phase features to better capture object extent.The experiments examine whether the distinct features are semantically consistent with object parts.Source: §6 · pp. 6–7
6.2. Results and Discussion
- 88.1% mAP for the first network and 82.6% for the second, a 5.5% performance reduction; the second highlights other important object parts.The reported phase scores are for Pascal VOC object location prediction.Source: Table 3 · p. 7, §6.2 · p. 7
6.1. Evaluation
- On 5,823 Pascal VOC 2012 main validation images, localization uses each per-class heat map’s maximal response and an 18-pixel tolerance; the two networks are evaluated separately.The maximal response is treated as the predicted location, and its confidence is used to compute AP.Source: §6.1 · p. 7, §6 · p. 7
6.2. Results and Discussion
- The two networks’ predicted locations are 69 pixels apart on average, and in most cases they focus on different image regions.The paper reports an average image size of 390 × 470 pixels and concludes that the different features can complement each other for object localization.Source: §6.2 · p. 7, §6.1 · p. 7
7. Conclusion
The conclusion frames incomplete object extent as a weakness of weakly supervised localization and presents two-phase learning as a way to mitigate it. The approach combines heat maps from both networks and is reported effective across three tasks on Pascal VOC 2012.
7. Conclusion
- Weakly supervised localization can miss object extent because networks focus on the most distinctive parts.The conclusion identifies this as an inherent weakness of the task.Source: §7 · p. 7
- Inference conditional feedback suppresses first-phase discriminative regions during retraining, and the two networks’ heat maps are combined to enhance localization.The paper proposes this two-phase procedure to reveal different features that remain within object regions.Source: §7 · p. 8
- Experiments on Pascal VOC 2012 report effectiveness for semantic segmentation, object saliency detection, and object location prediction.The conclusion describes the approach as mitigating the problem of incomplete object extent.Source: §7 · p. 8