Source-linked AI summary
Self-Supervised Difference Detection for Weakly-Supervised Semantic Segmentation
Wataru Shimoda, Keiji Yanai
TL;DR
Weakly supervised segmentation relies on visualization and mapping functions, but mapped results can contain noise and need not improve accuracy. SSDD predicts mask differences to select useful advice; on PASCAL VOC 2012, the paper reports 64.9% mean IoU on validation and 65.5% on test, described as new state-of-the-art performance.
Problem
Visualization results do not always match segmentation, and mapping functions such as CRF or iterative retraining do not guarantee accuracy improvements.
Source: §1 · p. 1Method
SSDD predicts differences between knowledge and mapped advice to estimate noise and select useful information from the mapping results.
Source: §1 · p. 2Results
64.9% mean IoU on the val set and 65.5% on the test set; the paper reports new state-of-the-art PASCAL VOC 2012 performance.
Source: §1 · p. 2Takeaways & Limitations
The method is reported effective for refining masks in both seed generation and segmentation training stages.
Source: §6 · p. 8Small-object regions tend to vanish in CRF, a tendency DD-Net also learns; refinement is also ineffective when both input masks fail to segment.
Source: §5.2 · p. 7
Abstract
from arXiv · showhide
To minimize the annotation costs associated with the training of semantic segmentation models, researchers have extensively investigated weakly-supervised segmentation approaches. In the current weakly-supervised segmentation methods, the most widely adopted approach is based on visualization. However, the visualization results are not generally equal to semantic segmentation. Therefore, to perform accurate semantic segmentation under the weakly supervised condition, it is necessary to consider the mapping functions that convert the visualization results into semantic segmentation. For such mapping functions, the conditional random field and iterative re-training using the outputs of a segmentation model are usually used. However, these methods do not always guarantee improvements in accuracy; therefore, if we apply these mapping functions iteratively multiple times, eventually the accuracy will not improve or will decrease. In this paper, to make the most of such mapping functions, we assume that the results of the mapping function include noise, and we improve the accuracy by removing noise. To achieve our aim, we propose the self-supervised difference detection module, which estimates noise from the results of the mapping functions by predicting the difference between the segmentation masks before and after the mapping. We verified the effectiveness of the proposed method by performing experiments on the PASCAL Visual Object Classes 2012 dataset, and we achieved 64.9\% in the val set and 65.5\% in the test set. Both of the results become new state-of-the-art under the same setting of weakly supervised semantic segmentation.
1. Introduction
Weakly supervised segmentation avoids costly pixel annotations, but visualization-to-segmentation mappings can be noisy and fail to improve accuracy. The paper proposes self-supervised difference detection to select useful mapped advice and applies it in two stages.
1. Introduction
- Image-level annotation is easier to obtain than pixel-level labels, but weakly supervised methods retain a large performance gap from fully supervised methods.Pixel-level annotation is costly to create.Source: §1 · p. 1
- Visualization results do not always match segmentation, and mapping functions such as CRF and iterative retraining do not guarantee accuracy improvements.The paper treats mapping outputs as supervision containing noise.Source: §1 · p. 1
- SSDD predicts differences between knowledge and advice without annotation, treating mask differences as self-supervision to estimate noise and select useful information.The paper calls mapping-function inputs knowledge and noisy mapped supervision advice.Source: §1 · pp. 1–2
- The selection premise is that advice contains enough correct information for predictable advice to correlate strongly with true advice.The method selects advice judged true by predicting it from difference-detection inference.Source: §1 · p. 2
- SSDD refines PSA's CRF results during seed generation and is also used inside the fully supervised segmentation training loop.The paper reports applying the module in both stages.Source: §1 · p. 2
- 64.9% mean IoU on the val set and 65.5% on the test set; the SSDD approach reports new state-of-the-art PASCAL VOC 2012 performance.The reported values are for the proposed method's PASCAL VOC 2012 evaluation.Source: §1 · p. 2
2. Related Works
Related work spans visualization-based localization, CRF refinement, pseudo-label training, and methods that use additional information. The paper positions its approach among these methods while using no additional information beyond its stated training inputs.
Visualization
- Early visualization methods used classification-related signals to localize objects, with CAM becoming widely adopted in weakly supervised segmentation.The passage discusses back-propagated derivatives, activation-based visualization, and global-pooling methods.Source: §2 · p. 2
Region refinement for WSS results using CRF
- CRF can refine ambiguous outlines using low-level image features, but it does not guarantee higher mean IoU; this work integrates initial and CRF-refined masks using confidence maps.The proposed integration is intended to prevent degradation from applying CRF.Source: §2 · p. 2
Training fully supervised segmentation model under weakly supervised setting
- Prior fully supervised segmentation training under weak supervision used image-level losses or saliency-derived pseudo-labels, with iterative retraining also reported to improve results.The cited iterative method reused segmentation outputs as new pixel-level annotations.Source: §2 · p. 2
Generating pixel-level labels during training of an FSS model
- CCNN and EM-adopt generate pixel labels from class labels and model outputs, while PSL regenerates seeds during training and DSRG expands seed regions.The methods use different constraints or seed-generation strategies to train segmentation models without manual pixel-level labels.Source: §2 · p. 3
WSS methods using additional information
- Other WSS methods use extra annotations, web images, or saliency; this paper states that it uses no additional information beyond VOC images with image-level labels and ImageNet-pretrained CNNs.The related-work passage lists boxes, points, scribbles, motion segmentation, web images, and saliency as additional information used by prior work.Source: §2 · p. 3
3. Method
The method treats mapping-function outputs as noisy advice and selects useful information by predicting where advice differs from the input knowledge. Its SSDD module learns difference masks without special annotation or additional data, then uses confidence maps to integrate masks.
3. Method
- SSDD selects information from mapping-function results treated as noisy advice by predicting differences between those results and the input knowledge.The method calls the mapping inputs “knowledge” and the mapped results “advice.”Source: §3 · p. 3
3.1. Difference detection network
- The difference detection network uses high- and low-level image features with a mask to predict a pixelwise difference map, trained without special annotation or additional data.Its architecture consists of three convolutional layers and one residual block.Source: §3.1 · pp. 3–4
3.2. Self-supervised difference detection module
- The SSDD module integrates masks using confidence maps, evaluating advice by how close its predicted confidence values are to those for knowledge.The method uses the knowledge-to-advice inference results to set sample-specific thresholds.Source: §3.2 · p. 4
- The refined mask is generated from the knowledge and advice masks, with a bias parameter controlling the selection threshold and enhancing categories present in the input image.The paper names this mask-generation processing flow the SSDD module.Source: §3.2 · p. 4
4. Introducing SSDD modules into the processing flow of WSS
The WSS processing flow applies SSDD first to PSA outputs and their CRF refinements, then during segmentation training to seed labels and model-generated masks. Dynamic refinement limits seed-label interpolation to predictable regions of difference.
4. Introducing SSDD modules into the processing flow of WSS
- The adopted WSS flow has two stages: static seed generation using PSA and CRF masks, followed by dynamic refinement using those seeds and segmentation-model outputs as advice.The method is presented as adaptable by treating mapping-function inputs as knowledge and mapping results as advice.Source: §4 · pp. 4–5
4.1. Seed mask generation stage with static region refinement
- Static refinement uses PSA probability maps as knowledge and their CRF results as advice, then applies SSDD to integrate the resulting masks.The paper excludes training samples when either input mask lacks correct labels, using differences in class-specific pixel counts to identify bad samples.Source: §4.1 · p. 5
4.2. Training stage of a fully supervised segmentation model with a dynamic region refinement
- Dynamic refinement constrains seed-label interpolation to regions whose differences between newly generated labels and seed labels are predictable.Each iteration generates a refined mask from the model and CRF outputs, then further refines it with the seed labels before training the segmentation model.Source: §4.2 · pp. 5–6
- Dynamic refinement excludes bad samples when input masks lack correct regions and adds a segmentation branch to address a local minimum that makes difference predictions constant.The added branch is trained to produce an intermediate mask between the seed and model-generated masks.Source: §4.2 · p. 6
5. Experiments
Experiments on PASCAL VOC 2012 evaluate SSDD in seed generation and segmentation training, finding improved results against PSA and other weakly supervised methods. The analysis also identifies settings where refinement is less effective.
5. Experiments
- Evaluation uses PASCAL VOC 2012, including augmented training data, and reports mean IoU on validation and test sets using the official evaluation server.The segmentation model uses ResNet-38, with its final upsampling rate set to 2 to reduce CRF computational cost.Source: §5 · p. 6, §5.1 · p. 6
5.2. Analysis of static region refinement
- Static refinement remains highly accurate across tested CRF parameter values, although weakening wrgb reduces the difference between CRF-only and SSDD-plus-CRF results.The authors suggest the method corrects partial CRF failures; small objects can nevertheless vanish, and refinement is ineffective when both input masks fail.Source: §5.2 · p. 7, Figure 6 · p. 7
5.3. Analysis of the whole proposed method
- Dynamic refinement outperforms PSA by more than 3.2 points, and its masks show more accurate object locations and boundaries than static refinement.The authors attribute the improvement over static refinement to end-to-end learning increasing the number of good advice.Source: §5.3 · p. 7
- Gains exceed 10% for cat, cow, horse, and sheep, but improvement depends on true advice exceeding false advice.The authors report little improvement or declines when that precondition is unmet, including for potted plant, airplane, and person.Source: §5.3 · p. 7
- The method achieves the highest scores among methods using the same supervision on PASCAL VOC 2012 validation and test data.The comparison reports higher performance than recent works including MEFF and TPL.Source: §5.3 · p. 8, Table 2 · p. 8
- Against methods using relatively cheap additional information, the proposed method outperforms all listed methods, though comparisons with some methods are not fully fair.The authors note differences in network, augmentation, and training epochs for comparisons with AISC and Shen et al.Source: §5.3 · p. 8, Table 3 · p. 8
6. Conclusions
The conclusion presents SSDD as a way to refine masks before and after refinement processes such as CRF. It reports effective use in two stages and improved PASCAL VOC 2012 results with three modules.
6. Conclusions
- Three SSDD modules achieved the best PASCAL VOC 2012 results among weakly supervised methods, with and without additional supervision.The conclusion attributes this result to the combined modules.Source: §6 · p. 8
- SSDD refines a segmentation mask using a pair of masks from before and after a refinement process such as CRF.The method is used for static region refinement during seed generation and dynamic region refinement during training.Source: §6 · p. 8
- The seed-generation stage refines PSA's CRF results, while the training stage refines masks produced by a segmentation model and CRF.The conclusion describes these as the two effective stages for SSDD.Source: §6 · p. 8
A.1 Details of the simple decision
The simple decision selects advice using difference-detection inference, but the decision can fail when that inference is too easy to distinguish true from false advice.
A.1 Details of the simple decision
- The confidence score measures how close dK is to dA, and the method ignores advice when the difference is large enough.The decision selects advice based on the inference results of difference detection.Source: §A.1 · p. 11
- When difference detection is too easy, dK can approach dA for false advice, making it indistinguishable from true advice.The passage gives dK = 1 and dA = 1 as a case where the method cannot distinguish true from false advice.Source: §A.1 · p. 11
- The method excludes typical failed-advice samples when per-class pixel differences in the masks are obviously large.These exclusions are intended to keep the differences between dK and dA large when inferring bad advice.Source: §A.1 · p. 11
- The bad-sample rule uses a 0.5 threshold with C as the input image's image-level label set; the threshold was chosen empirically.The displayed condition is given as c | < 0.5.Source: §A.1 · p. 11
A.2 Details of the bias in Eq.(3)
The bias section distinguishes a bias for differences between knowledge and advice from a class-category bias, and uses the latter to prioritize a class label when advice appears to have failed.
A.2 Details of the bias in Eq.(3)
- bdd is a bias for the difference between knowledge and advice, while bclass is a bias for the class category.The section introduces these bias values in relation to Eq.(3) and Eq.(17).Source: §A.2 · p. 11
- When per-class mask differences are obviously large, bclass prioritizes that class's label over difference-detection results.The passage treats this condition as evidence that advice has failed.Source: §A.2 · p. 11
- The values of bdd and bclass were selected using grid search.The section defines the biases as values used in the decision rule.Source: §A.2 · p. 11
A.3 Values of hyperparameters
The hyperparameter experiments report the best mean IoU at bdd = 0.4 and bclass = 1.0, with α = 0.5 scoring best among the tested loss coefficients.
A.3 Values of hyperparameters
- α = 0.5, the balanced loss-coefficient setting, achieved the best mean IoU score.The authors report that α was not critical to changes in mean IoU.Source: §A.3 · p. 11
- The hyperparameter experiments varied bias values and loss coefficients and measured mean IoU.Table A-1 is described as reporting results with different parameters.Source: §A.3 · p. 11, Table A-1 · p. 11
- bdd = 0.4 produced the maximum mean IoU, and bclass = 1.0 produced the maximum mean IoU for the class bias.The authors suggest class biases for missing categories may help training robustness.Source: Table A-1 · p. 11, §A.3 · p. 11
A.4 Detailed comparison with existing works on the PASCAL VOC 2012 val and test sets
The section presents detailed comparisons with existing works on PASCAL VOC 2012 across validation and test sets, with and without additional supervision.
A.4 Detailed comparison with existing works on the PASCAL VOC 2012 val and test sets
- The comparison includes PASCAL VOC 2012 validation results without additional supervision.Table A-2 covers the validation set under this setting.Source: Table A-2 · p. 12
- The comparison includes PASCAL VOC 2012 test results without additional supervision.Table A-3 covers the test set under this setting.Source: Table A-3 · p. 12
- The comparison also includes PASCAL VOC 2012 validation results with additional supervision.Table A-4 covers the validation set under this setting.Source: Table A-4 · p. 12