Source-linked AI summary
A Simple Baseline for Semi-supervised Semantic Segmentation with Strong Data Augmentation
Jianlong Yuan, Yifan Liu, Chunhua Shen, Zhibin Wang, Hao Li
TL;DR
The paper tackles the expense of pixel-level annotation by proposing a semi-supervised semantic-segmentation framework. It combines strong augmentation with distribution-specific batch normalization and a self-correction loss, achieving state-of-the-art performance on Cityscapes and Pascal VOC.
Problem
Pixel-level annotation for semantic segmentation is expensive and time-consuming, creating a need to use large amounts of unlabeled data with limited labeled data.
Method
The framework combines strong augmentation, distribution-specific batch normalization, pseudo-label learning, and a self-correction loss for noisy pseudo-labels.
Results
The approach achieves state-of-the-art performance on the Cityscapes and Pascal VOC datasets in semi-supervised settings.
Takeaways & Limitations
Strong augmentation can be used effectively for semi-supervised semantic segmentation when batch-normalization distribution shifts and pseudo-label noise are addressed.
Abstract
from arXiv · showhide
Recently, significant progress has been made on semantic segmentation. However, the success of supervised semantic segmentation typically relies on a large amount of labelled data, which is time-consuming and costly to obtain. Inspired by the success of semi-supervised learning methods in image classification, here we propose a simple yet effective semi-supervised learning framework for semantic segmentation. We demonstrate that the devil is in the details: a set of simple design and training techniques can collectively improve the performance of semi-supervised semantic segmentation significantly. Previous works [3, 27] fail to employ strong augmentation in pseudo label learning efficiently, as the large distribution change caused by strong augmentation harms the batch normalisation statistics. We design a new batch normalisation, namely distribution-specific batch normalisation (DSBN) to address this problem and demonstrate the importance of strong augmentation for semantic segmentation. Moreover, we design a self correction loss which is effective in noise resistance. We conduct a series of ablation studies to show the effectiveness of each component. Our method achieves state-of-the-art results in the semi-supervised settings on the Cityscapes and Pascal VOC datasets.
1. Introduction
The paper addresses the cost of pixel-level labels by proposing a semi-supervised segmentation framework that combines strong augmentation, distribution-specific batch normalization, and noise-resistant self-training. It reports state-of-the-art performance on Cityscapes and Pascal VOC.
- Motivation: Pixel-level annotation is costly and time-consuming, motivating semi-supervised semantic segmentation that uses unlabeled data alongside limited labels.Semantic segmentation requires labeling every pixel, which can be 60 times more expensive than image-level annotation.
- Contributions: The method applies strong augmentation during semi-supervised training to make better use of unlabeled data without modifying the network structure.This design is motivated by consistency-learning methods that use strong and weak augmentations.
- Motivation: Strong augmentation can shift batch-normalization statistics and create a domain gap between training and test distributions.The paper links this mismatch to the changed mean and variance distributions produced by strong augmentation.
- Results: The proposed approach achieves state-of-the-art performance on Cityscapes and Pascal VOC under semi-supervised settings.The paper also reports quantitative and qualitative comparisons against recent semi-supervised approaches.
- Contributions: The framework uses distribution-specific batch normalization to accommodate augmentation-induced distribution discrepancies and a self-correction loss to reduce pseudo-label noise.The self-correction loss dynamically modifies pixel weights and learning targets for unreliable pseudo-label regions.
2. Our Method
The framework trains a teacher on labeled data, generates pseudo-labels for unlabeled images, and trains a strongly augmented student. It addresses augmentation-induced batch-statistics shifts with distribution-specific normalization and pseudo-label noise with self-correction loss.
- 2.1. Semi-supervised Semantic Segmentation: The framework trains an initial teacher on labeled images, generates hard pseudo-labels for unlabeled images, and trains a student with strong augmentation.It is designed to apply to existing segmentation networks without auxiliary sub-networks.
- 2.2. Distribution-Specific Batch Normalization: Distribution-specific batch normalization routes strongly augmented images through a separate BN branch while weakly augmented images use standard normalization, discarding the strong-augmentation branch at test time.This accommodates the distribution discrepancy introduced within a training batch.
- 2.2. Distribution-Specific Batch Normalization: Strong augmentation shifts activation distributions, so batch normalization must account for statistics differing between strongly and weakly augmented samples.Standard batch normalization uses mini-batch statistics during training and moving statistics during testing.
- 2.3. Self-correction Loss: Pseudo-labels can contain unreliable regions because the teacher has limited capacity, making direct learning from every pixel vulnerable to label noise.The challenge is especially relevant because semantic segmentation performs classification independently at each pixel.
- 2.3. Self-correction Loss: Self-correction loss dynamically weights pixels by prediction confidence and applies reverse learning to low-confidence pseudo-labels to reduce overfitting to label noise.The dynamic weight is the largest softmax activation across classes, and the loss is applied only to pseudo-labels.
3. Experiments
Experiments show that the framework benefits most when labeled data are scarce, strong augmentation is paired with DSBN and self-correction, and the resulting method performs favorably on Cityscapes and Pascal VOC.
- Different semi-supervised settings: 5.2%, 4.4%, 1.8%, and 1.8% improvements are reported across the evaluated semi-supervised settings, with larger gains when fewer labeled images are available.Using half the labeled data with the framework reaches the same 78.7% mIoU as supervised learning with all labeled training data.
- Different semi-supervised settings: Increasing the unlabeled-to-labeled ratio improves results, but gains gradually flatten because the initial teacher model has limited capacity.The labeled set is fixed at one-eighth of the training set in this experiment.
- Impact of the strong augmentation: Applying strong augmentation to unlabeled data with pseudo-labels improves mIoU by 0.8%, while self-correction keeps training relatively stable instead of diverging quickly.The experiments also report that DSBN contributes to training, and qualitative results show improved separation of large and small objects.
- Comparison with State-of-the-art Methods: 74.1%, 77.8%, and 78.7% mIoU are achieved with DeepLabV3Plus on Cityscapes, performing favorably against previous state-of-the-art methods.With DeepLabV2, the framework achieves 67.6%, 69.3%, and 70.7%.
- Comparison with State-of-the-art Methods: 79.3% mIoU with Xception-65 on Pascal VOC outperforms PseudoSeg by 1.8%, while matching the fully supervised setting.The framework also achieves 75.0% mIoU with ResNet-101.
4. Additional Ablation Study with DeepLabV2
The DeepLabV2 ablation study evaluates proposed modules under a 1/8 labeled-image setting and examines performance across training iterations.
- The ablation study evaluates the proposed modules with DeepLabV2 based on ResNet-101 under a 1/8 labeled-image setting.The baseline uses self-training, with pseudo-labels generated by a teacher model for unlabeled data.
- The main ablation results are reported in Table 6 for the proposed semi-supervised learning framework.
- More iterations generally improve performance, but the gains gradually flatten over time.The iteration analysis uses the DeepLabV2 model.
5. Conclusion
The paper presents a semi-supervised segmentation framework combining strong augmentation, DSBN, pseudo-label learning, and self-correction. Quantitative and qualitative comparisons report favorable performance against recent state-of-the-art semi-supervised approaches.
- The framework combines strong augmentation, distribution-specific batch normalization, pseudo-label learning, and a self-correction loss for semantic segmentation.DSBN addresses batch-normalization statistics shifts, while self-correction mitigates pseudo-label noise.
- Quantitative and qualitative comparisons show favorable performance against recent state-of-the-art semi-supervised approaches.