Source-linked AI summary
Perturbed and Strict Mean Teachers for Semi-supervised Semantic Segmentation
Yuyuan Liu, Yu Tian, Yuanhong Chen, Fengbei Liu, Vasileios Belagiannis, Gustavo Carneiro
TL;DR
Semi-supervised semantic segmentation must extract useful training signal from unlabelled images despite costly pixel-level annotation and inaccurate predictions. The paper extends mean teachers with stricter confidence-weighted learning and multiple perturbations, achieving the best reported results on Pascal VOC 2012 and Cityscapes.
Problem
Semi-supervised semantic segmentation seeks useful training signal from many unlabelled images because pixel-level labelling is costly and time-consuming.
Method
The method adds an auxiliary teacher, replaces MT's MSE with confidence-weighted CE, and combines input, feature, and network perturbations including T-VAT.
Results
The approach achieves the best performance on Pascal VOC 2012 and Cityscapes, and exceeds CPS by around 1% to 2% across Pascal VOC partition protocols.
Takeaways & Limitations
More accurate teacher predictions support stricter confidence-based training and challenging perturbations for improved consistency-learning generalisation.
Abstract
from arXiv · showhide
Consistency learning using input image, feature, or network perturbations has shown remarkable results in semi-supervised semantic segmentation, but this approach can be seriously affected by inaccurate predictions of unlabelled training images. There are two consequences of these inaccurate predictions: 1) the training based on the "strict" cross-entropy (CE) loss can easily overfit prediction mistakes, leading to confirmation bias; and 2) the perturbations applied to these inaccurate predictions will use potentially erroneous predictions as training signals, degrading consistency learning. In this paper, we address the prediction accuracy problem of consistency learning methods with novel extensions of the mean-teacher (MT) model, which include a new auxiliary teacher, and the replacement of MT's mean square error (MSE) by a stricter confidence-weighted cross-entropy (Conf-CE) loss. The accurate prediction by this model allows us to use a challenging combination of network, input data and feature perturbations to improve the consistency learning generalisation, where the feature perturbations consist of a new adversarial perturbation. Results on public benchmarks show that our approach achieves remarkable improvements over the previous SOTA methods in the field. Our code is available at https://github.com/yyliu01/PS-MT.
1. Introduction
Semi-supervised semantic segmentation seeks useful training signal from abundant unlabelled images because pixel-level annotation is costly. This paper targets inaccurate consistency predictions with stricter mean teachers, multiple perturbations, and T-VAT, reporting best performance on Pascal VOC 2012 and Cityscapes.
- Pixel-level labelling is costly and time-consuming, motivating semi-supervised segmentation with small labelled and large unlabelled image sets.
- Consistency learning enforces agreement across perturbed views, but inaccurate predictions can mutually degrade network-perturbation training and cause confirmation bias under strict CE loss.
- The method extends mean teachers with an auxiliary teacher and replaces MT’s MSE with confidence-weighted CE to improve unlabelled-image prediction accuracy and training convergence.
- The approach combines network, feature, and input-image perturbations, including the adversarial T-VAT feature perturbation, to improve consistency-learning generalisation.
- The evaluation reports best performance on Pascal VOC 2012 and Cityscapes among the compared approaches.
2. Related Work
Related work frames semi-supervised segmentation through supervised architectures, SSL assumptions, pseudo-labelling, and consistency learning. Existing consistency methods perturb inputs, features, or networks, while this paper motivates more accurate and targeted perturbations.
- Supervised semantic segmentation: Supervised segmentation classifies every image pixel into a closed set of visual classes using architectures such as FCNs, DeepLabV3+, and PSPNet.
- Semi-supervised learning: Semi-supervised learning trains models with labelled and unlabelled images under assumptions including smoothness.
- Consistency learning: The approach uses weak augmentation for mean teachers and strong augmentation for the student, with teacher predictions supporting consistency training.
- Consistency learning: Consistency methods obtain multiple views through input augmentation, feature noise injection, or differently initialised networks whose predictions are encouraged to agree.
- Consistency learning: Existing feature perturbations can be effective, but the paper motivates more targeted adversarial noise because perturbation methods depend on accurate segmentation predictions.
3. Method
The method extends mean-teacher training with an auxiliary teacher, confidence-weighted CE consistency learning, and multiple perturbations. It combines network, adversarial feature, and input-image perturbations while training the student with supervised and consistency losses.
- 3.1. Multiple Mean Teachers and Student Models: The model uses two mean teachers and one student with the same encoder-decoder network structure, improving unlabelled-image prediction through double ensembling.Each teacher represents a temporal ensemble of the student model, and only one teacher is updated per epoch.
- 3.2. Training with Multiple Perturbations and a Strict Confidence-weighted CE Loss: The method applies network, feature, and input-image perturbations to strengthen consistency learning.Input perturbations include weak-strong augmentation pairs, CutMix, and Zoom In/Out.
- 3.2. Training with Multiple Perturbations and a Strict Confidence-weighted CE Loss: The full student objective combines supervised segmentation loss on labelled data with confidence-weighted CE consistency loss on unlabelled data.The consistency term is weighted by β and uses teacher-derived confidence values at each pixel.
- 3.2. Training with Multiple Perturbations and a Strict Confidence-weighted CE Loss: Teacher predictions provide hard pseudo-labels and confidence weights, with confidence enabled only when the ensemble prediction exceeds threshold τ.The teacher ensemble averages the two teachers’ logits before applying softmax; the student is trained against the resulting pseudo-labels.
- 3.2. Training with Multiple Perturbations and a Strict Confidence-weighted CE Loss: T-VAT estimates adversarial feature noise from the teacher ensemble and applies it to student features, pushing representations toward classification boundaries.The perturbation maximizes prediction divergence subject to an L2-norm constraint ||r_adv||_2 <= ε.
- 3.2. Training with Multiple Perturbations and a Strict Confidence-weighted CE Loss: The method replaces the usual L2 consistency loss with a stricter CE-based loss because L2 can converge poorly and produce vanishing gradients.This replacement relies on the extended mean-teacher model producing reliable segmentation predictions.
4. Experiments
The experiments evaluate the approach across datasets, architectures, labelled-data partitions, baselines, and component ablations. Results consistently report improvements over supervised and prior SOTA methods, while ablations isolate the contributions of Conf-CE, T-VAT, the auxiliary teacher, and perturbation placement.
- Experimental setting: Experiments use Pascal VOC 2012 and Cityscapes, primarily with DeepLabV3+, while PSPNet tests architectural generalisation.Pascal VOC 2012 provides 1,464 official labelled images; the experiments also use partition protocols and additional unlabelled data.
- Results on partition protocols: 6.01% and 5.97% mIoU gains over fully supervised baselines occur on Pascal VOC 2012 at the 1/16 labelled ratio with ResNet50 and ResNet101.The approach also improves by 2% to 5% at larger Pascal VOC partitions and by 2% and 6% on Cityscapes across protocols.
- Comparison with SOTA: The approach achieves the best Pascal VOC 2012 results across partition protocols and exceeds CPS by around 1% to 2%.On Cityscapes, it outperforms CAC by nearly 4.6% and 2.4% at the 1/8 and 1/4 partitions, respectively.
- Official labelled set: 5.01% mIoU separates the approach from Yuan et al.’s self-training single network, while official-label experiments also outperform SOTA methods for PSPNet and DeepLabV3+.For PSPNet, gains over DARS and CCT are 0.7% and 5.19% mIoU, respectively.
- Few-supervision study: 4.86% mIoU separates the approach from CPS with 366 labelled Pascal VOC 2012 images, and it achieves the best results across all few-supervision cases.The study uses 1/2, 1/4, 1/8, and 1/16 labelled partitions, with remaining data combined with an augmented set.
- Ablation study: Replacing MSE with Conf-CE improves mIoU by 2.30% and 2.81%, T-VAT adds nearly 1%, and the auxiliary teacher adds 0.83% and 0.84% for the two backbones.Applying CutMix after prediction improves mIoU by around 3%; adding CAM loss further boosts performance by 0.63% and 1.18%.
5. Conclusion
The paper introduces a consistency-based semi-supervised semantic segmentation method built on multiple mean teachers, a student network, and stricter confidence-weighted CE. Its accurate predictions enable combined perturbations, including the new adversarial T-VAT feature perturbation, and yield best results on Pascal VOC 2012 and Cityscapes.
- The proposed model uses multiple mean teachers and a student network to improve predictions for unlabelled images.The auxiliary teacher supports more accurate consistency-learning targets.
- It replaces the original MT MSE loss with a stricter confidence-weighted CE loss called Conf-CE.The authors note that Conf-CE may still overfit remaining prediction mistakes.
- More accurate predictions allow the method to combine network, feature, and input image perturbations for better generalisation.
- T-VAT is a new adversarial feature perturbation that further improves the approach’s generalisation.
- The method outperforms previous methods on Pascal VOC 2012 and Cityscapes, becoming the new SOTA for semi-supervised semantic segmentation.