Source-linked AI summary
ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation
Lihe Yang, Wei Zhuo, Lei Qi, Yinghuan Shi, Yang Gao
TL;DR
Semi-supervised semantic segmentation must exploit unlabeled images despite noisy pseudo labels and teacher–student coupling. The paper develops ST with strong augmentation and ST++ with stability-based selective retraining, and reports that both outperform prior methods across datasets and settings.
Problem
Semi-supervised semantic segmentation needs effective ways to utilize abundant unlabeled images when dense pixel-wise annotation is costly or infeasible.
Method
ST injects strong data augmentations on unlabeled images, while ST++ selectively retrains on reliable images ranked by holistic prediction stability.
Results
ST and ST++ outperform previous methods across extensive datasets, settings, and architectures with few hyper-parameters.
Takeaways & Limitations
Strong augmentation makes plain self-training a strong segmentation baseline, while selective image-level retraining further advances it.
Abstract
from arXiv · showhide
Self-training via pseudo labeling is a conventional, simple, and popular pipeline to leverage unlabeled data. In this work, we first construct a strong baseline of self-training (namely ST) for semi-supervised semantic segmentation via injecting strong data augmentations (SDA) on unlabeled images to alleviate overfitting noisy labels as well as decouple similar predictions between the teacher and student. With this simple mechanism, our ST outperforms all existing methods without any bells and whistles, e.g., iterative re-training. Inspired by the impressive results, we thoroughly investigate the SDA and provide some empirical analysis. Nevertheless, incorrect pseudo labels are still prone to accumulate and degrade the performance. To this end, we further propose an advanced self-training framework (namely ST++), that performs selective re-training via prioritizing reliable unlabeled images based on holistic prediction-level stability. Concretely, several model checkpoints are saved in the first stage supervised training, and the discrepancy of their predictions on the unlabeled image serves as a measurement for reliability. Our image-level selection offers holistic contextual information for learning. We demonstrate that it is more suitable for segmentation than common pixel-wise selection. As a result, ST++ further boosts the performance of our ST. Code is available at https://github.com/LiheYoung/ST-PlusPlus.
1. Introduction
The paper revisits self-training for semi-supervised semantic segmentation, strengthening it with strong data augmentation and selective image-level retraining. ST and ST++ outperform prior methods across datasets and settings, especially with scarce labels.
- Motivation: Semi-supervised segmentation reduces costly pixel-wise annotation by learning from few labeled and abundant unlabeled images.The paper motivates this setting because dense manual labeling is costly, laborious, and sometimes infeasible.
- Motivation: The paper asks whether delicate semi-supervised mechanisms are necessary and whether classical self-training remains competitive for segmentation.It presents two simple techniques intended to restore self-training as a strong competitor.
- ST baseline: Strong data augmentations on unlabeled images decouple teacher and student predictions while alleviating overfitting to noisy pseudo labels.The approach uses strong augmentations during self-training without iterative retraining, manual confidence thresholds, or repeated minibatch pseudo-label generation.
- ST++: ST++ selectively retrains using reliable unlabeled images ranked by holistic prediction-level stability across the training course.The paper argues that image-level selection provides contextual regions more suitable for segmentation than pixel-wise selection.
- Results: ST and ST++ outperform previous methods across extensive settings and architectures on Pascal and Cityscapes with few hyper-parameters.The paper also reports particularly strong gains in the extremely scarce-label regime, including a setting with 92 labels.
2. Related Work
The related work situates ST++ among consistency regularization, entropy minimization, self-training, and uncertainty estimation methods. Unlike approaches using complex mechanisms or pixel-level confidence filtering, it combines strong augmentation with image-level selective retraining.
- Semi-supervised learning: Semi-supervised learning methods commonly follow consistency regularization or entropy minimization, with consistency enforcing stable predictions under perturbations.Earlier methods also use saved checkpoints or teacher models based on exponential moving averages.
- Semi-supervised semantic segmentation: Segmentation methods have used GANs, perturbed embeddings, contextual crops, dual models, and weak-to-strong consistency with calibration.The paper contrasts these mechanisms with a plain self-training pipeline coupled with strong data augmentations.
- Selective retraining: ST++ progressively selects reliable images for retraining and then uses the improved student to generate labels for less reliable images.Its pseudocode includes checkpoint-based scoring, staged dataset construction, student reinitialization, and successive retraining.
- Positioning: Unlike prior augmentation work using task-specific designs or plain training, ST++ applies common recognition augmentations within a curriculum-like selective retraining framework.The related-work discussion emphasizes the combination of strong augmentation and safer exploitation of unlabeled data.
- Uncertainty estimation: ST estimates image-level uncertainty from holistic prediction stability across evolving pseudo masks rather than filtering uncertain pixels with a fixed confidence threshold.This distinguishes it from methods such as FixMatch and DMT.
3. Method
The method formalizes semi-supervised segmentation around supervised learning, pseudo-labeling, and student retraining, then strengthens self-training with strong augmentation and selective image-level retraining. ST++ prioritizes unlabeled images using prediction stability measured across supervised-training checkpoints.
- Problem definition: Semi-supervised segmentation combines densely labeled images with abundant unlabeled images, optimizing supervised and unsupervised losses with λ controlling their tradeoff.The supervised loss is typically cross-entropy, while the unsupervised loss distinguishes methods.
- Plainest self-training scheme: Plain self-training trains teacher T on labeled data, predicts hard pseudo labels for unlabeled images, and retrains student S on both datasets.The scheme uses three steps and does not require iterative training.
- ST: Inject SDA on unlabeled images: Strong data augmentations on unlabeled images make student optimization more challenging, helping address noisy-label overfitting and teacher–student prediction coupling.The augmentations are applied during retraining and are distinct from weak or basic augmentations.
- ST: Inject SDA on unlabeled images: ST simplifies labeled–unlabeled balancing by oversampling labeled data to roughly the unlabeled-data scale and sampling uniformly without extra hyperparameters.The resulting learner is optimized in a fully supervised fashion, and unlabeled images are annotated only once.
- ST++: Select and prioritize reliable images: ST++ selectively retrains on reliable images, using pseudo-mask stability across checkpoints to rank samples before relabeling the remaining images.Stability is estimated through meanIOU between earlier checkpoint masks and the final checkpoint mask; the top R images enter the first retraining phase.
4. Experiments
Across Pascal VOC and Cityscapes settings, ST consistently outperforms prior methods, while ST++ further improves performance through selective re-training. Ablations attribute these gains to strong augmentations on unlabeled data and reliable image-level selection.
- Comparison with State-of-the-Art Methods: ST outperforms existing methods across four network architectures on Pascal VOC, and ST++ consistently boosts performance further.The comparison also reports a significant margin over supervised-only training.
- Ablation Studies: 73.2 vs. 73.3 shows that applying Cutout to labeled images is inferior to using it as strong regularization for unlabeled data.The result distinguishes SDA's role from a generic supervised-training trick.
- Comparison with State-of-the-Art Methods: ST and ST++ surpass prior state-of-the-art methods on Pascal VOC with original-training-set labels and on Cityscapes across labeled-data regimes.On Pascal VOC, both methods even outperform the fully supervised setting with only 1464 labeled images; Cityscapes evaluations range from 744 to 100 labeled images.
- Ablation Studies: Without SDA, self-training performance degrades across all settings, whereas applying SDA to labeled images negatively affects results.The ablation supports using strong perturbations specifically on unlabeled images.
- Ablation Studies: Reliable images have pseudo-mask meanIOU at least 15% higher than unreliable images, and re-labeling the latter with a better student markedly improves their masks.The result supports prioritizing reliable images during selective re-training.
- Ablation Studies: Image-level selective re-training consistently outperforms one-stage and pixel-level alternatives, while the first ST++ stage already reaches competitive ST results.The default 50% reliable-image selection is effective, 75% is slightly better, and an extra iterative stage improves results from 65.2% to 68.3% and 74.6% to 75.2% in scarce-label regimes.
5. Conclusion
The work establishes strong self-training frameworks for semi-supervised semantic segmentation and supports their effectiveness through extensive experiments and component analysis.
- Strong data augmentations make self-training a competitive baseline by addressing noisy-label overfitting and teacher–student prediction coupling.
- ST++ progressively leverages unlabeled images through an advanced framework that prioritizes their use during training.
- Across varied benchmarks and settings, both ST and ST++ outperform previous methods by a large margin.
- Detailed component experiments provide empirical analysis of the framework’s effectiveness.
- The framework is positioned as a strong baseline or competitor for future semi-supervised semantic segmentation research.