Source-linked AI summary
Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision
Xiaokang Chen, Yuhui Yuan, Gang Zeng, Jingdong Wang
TL;DR
Semi-supervised semantic segmentation seeks to exploit unlabeled images because pixel-level annotation is costly. The paper proposes cross pseudo supervision, in which two differently initialized networks use each other’s pseudo one-hot maps for consistency training and data expansion. Experiments report state-of-the-art performance on Cityscapes and PASCAL VOC 2012, including gains over CutMix-Seg under a 1/16 partition protocol.
Problem
Pixel-level manual annotation makes semantic segmentation data costly, motivating methods that learn from labeled and additional unlabeled images.
Method
Cross pseudo supervision trains two same-structure, differently initialized networks, using each network’s pseudo one-hot segmentation map to supervise the other.
Results
Cross pseudo supervision achieves state-of-the-art semi-supervised segmentation performance on Cityscapes and PASCAL VOC 2012.
Takeaways & Limitations
The approach combines cross-network prediction consistency with pseudo-labeled unlabeled data that expands training and improves segmentation training quality.
Takeaways & Limitations
Cityscapes comparisons do not include CutMix-Seg because its official implementation supports only single-GPU training and is infeasible with DeepLabv3+ under the available GPU memory limit.
Abstract
from arXiv · showhide
In this paper, we study the semi-supervised semantic segmentation problem via exploring both labeled data and extra unlabeled data. We propose a novel consistency regularization approach, called cross pseudo supervision (CPS). Our approach imposes the consistency on two segmentation networks perturbed with different initialization for the same input image. The pseudo one-hot label map, output from one perturbed segmentation network, is used to supervise the other segmentation network with the standard cross-entropy loss, and vice versa. The CPS consistency has two roles: encourage high similarity between the predictions of two perturbed networks for the same input image, and expand training data by using the unlabeled data with pseudo labels. Experiment results show that our approach achieves the state-of-the-art semi-supervised segmentation performance on Cityscapes and PASCAL VOC 2012. Code is available at https://git.io/CPS.
1. Introduction
The paper frames semi-supervised semantic segmentation as a response to costly pixel-level annotation and introduces cross pseudo supervision, which uses two differently initialized networks to supervise each other with pseudo segmentation maps. The method encourages prediction consistency while using pseudo-labeled data to expand training, achieving state-of-the-art performance on Cityscapes and PASCAL VOC 2012.
- Pixel-level manual labeling makes semantic segmentation data more expensive to obtain than data for image classification and object detection.
- Cross pseudo supervision feeds labeled and unlabeled images into two same-structure networks with different initializations.
- Each network uses the other network’s pseudo segmentation map as an additional supervisory signal, enforcing cross-network consistency.
- Pseudo-labeled data expands the effective training set, while later-stage pseudo segmentation becomes more accurate than supervised training using labeled data alone.
- Cross pseudo supervision achieves state-of-the-art semi-supervised segmentation performance on Cityscapes and PASCAL VOC 2012.
2. Related work
Prior semi-supervised segmentation work combines consistency regularization under perturbations with self-training from pseudo labels. This paper situates cross pseudo supervision among input, feature, and network perturbation methods while distinguishing its cross-network pseudo-label supervision.
- Consistency regularization: Consistency regularization aligns predictions or intermediate features across perturbed inputs, features, or networks.
- Consistency regularization: Input perturbation methods augment images and constrain predictions so the decision function lies in low-density regions.
- Consistency regularization: Feature perturbation uses multiple decoders to enforce consistency between their outputs, while GCT uses differently initialized networks.
- Self-training: Self-training retrains segmentation models using pseudo maps generated on unlabeled data by models previously trained on labeled data.
- Related approaches: PseudoSeg follows FixMatch by using weakly augmented-image pseudo segmentation to supervise strongly augmented images, whereas this paper uses two networks that supervise each other.
3. Approach
Cross pseudo supervision trains two differently initialized, structurally identical segmentation networks on labeled and unlabeled images, using each network’s pseudo labels to supervise the other. Its objective combines labeled-data supervision with bidirectional cross pseudo supervision, and it also supports CutMix augmentation.
- Cross pseudo supervision: Two parallel segmentation networks share the same structure but are initialized with different weights and process identically augmented inputs.Their labeled outputs are separately supervised by the corresponding ground-truth segmentation maps.
- Cross pseudo supervision: Each network converts its confidence map into a predicted one-hot pseudo segmentation map for cross-network supervision.At each pixel, the pseudo-label vector is computed from the corresponding confidence vector.
- Cross pseudo supervision: The bidirectional CPS loss uses one network’s pseudo label map to supervise the other network’s pixel-wise confidence map, and vice versa.The loss is applied on unlabeled data and defined analogously on labeled data.
- Training objective: The training objective combines the standard labeled-data supervision loss with the cross pseudo supervision loss using a trade-off weight λ.The supervision loss is standard pixel-wise cross-entropy over labeled images for both networks.
- CutMix augmentation: CutMix augmentation feeds CutMixed images to both networks and mixes source-image pseudo maps to supervise the other network.The mixed pseudo segmentation maps are generated from the two source images and used as supervision for the CutMixed output.
4. Discussions
The discussion distinguishes CPS from probability consistency, mean teacher, single-network pseudo supervision, and PseudoSeg-style approaches. It emphasizes CPS’s consistency objective together with its use of pseudo-labeled unlabeled data, while reporting stronger empirical performance than several alternatives.
- Cross probability consistency: CPS seeks consistency between two perturbed segmentation networks while additionally augmenting training data with unlabeled images and pseudo labels.The paper reports that cross pseudo supervision outperforms cross probability consistency in Table 4.
- Mean teacher: Unlike mean teacher, CPS uses two differently initialized networks and cross pseudo-label supervision rather than a moving-average teacher with no backpropagation.Mean teacher aligns student and teacher probability maps under different augmentations.
- PseudoSeg: The PseudoSeg-related comparison uses pseudo-label supervision between streams, with the cited structure differing in whether the inputs use weak and strong augmentations.The discussion identifies this as a structural comparison rather than reporting a result in the supplied passage.
- Single-network pseudo supervision: Single-network pseudo supervision performs poorly because same-network pseudo labels can make the network approximate its own labels and converge in the wrong direction.The paper contrasts this with cross-network pseudo labels, which differ because of network perturbation.
- Single-network pseudo supervision: CPS performs better than the CutMix-based single-network variant, indicating that network perturbation remains helpful alongside CutMix augmentation.The comparison is reported as evidence that the two perturbations provide complementary benefit.
5. Experiments
Across Cityscapes and PASCAL VOC 2012, cross pseudo supervision consistently outperforms supervised and semi-supervised baselines, with stronger gains often appearing when labeled data are scarce. Ablations further show benefits from CutMix, unlabeled-set supervision, and combining CPS with self-training.
- Cityscapes: 4.89%, 4.07%, 2.74%, and 2.42% are the gains over supervised training under Cityscapes 1/16, 1/8, 1/4, and 1/2 partitions with ResNet-50.Without CutMix, CPS consistently improves the supervised baseline across all four partition protocols.
- Cityscapes: 3.70%, 3.52%, 2.11%, and 2.02% are the corresponding gains with ResNet-101 under Cityscapes 1/16, 1/8, 1/4, and 1/2 partitions.The method also achieves the best performance among partition protocols with both ResNet-50 and ResNet-101 backbones.
- PASCAL VOC 2012: On PASCAL VOC 2012, CPS with CutMix achieves the best performance under all partition protocols and exceeds CutMix-Seg by 3.08% with ResNet-50 and 1.92% with ResNet-101 at 1/16.Without CutMix, CPS consistently outperforms other methods except CutMix-Seg.
- Augmentation: CutMix is more important with fewer labeled images, yielding a 3.77% gain at 1/16 versus 0.47% at 1/8 with ResNet-50.On Cityscapes ResNet-101, its extra gains are 4.22%, 1.91%, and 0.13% at 1/16, 1/8, and 1/2 partitions.
- Cityscapes: 80.08% mIoU under the Cityscapes 1/2 partition with ResNet-101 exceeds GCT by 1.50%.This result is reported for the method with CutMix augmentation.
- Ablations: Cross pseudo supervision on unlabeled data improves more than its labeled-set counterpart in most cases, while using both is overall best.With ResNet-50, unlabeled-set CPS improves the baseline by 3.57% on PASCAL VOC 2012 and 4.07% on Cityscapes, versus 0.56% and 1.41% for labeled-set CPS.
- Ablations: CPS outperforms cross probability consistency by 2.36% with ResNet-50 and 1.94% with ResNet-101 on Cityscapes when applied to labeled and unlabeled sets.The comparison supports using pseudo segmentation maps rather than only aligning probability vectors.
- Ablations: Combining CPS with conventional self-training outperforms either method alone, indicating complementary behavior.The comparison accounts for self-training’s multi-stage optimization by also evaluating a longer-trained CPS variant.
6. Conclusion
Cross pseudo supervision is a simple semi-supervised segmentation approach that enforces consistency between differently initialized networks and uses pseudo-labeled unlabeled data to improve performance.
- CPS uses two networks with the same structure but different initialization, using each network’s one-hot pseudo segmentation map to supervise the other.
- Pseudo-labeled unlabeled data expands the training set and improves segmentation performance as pseudo segmentation becomes more accurate later in training.
A. More Implementation Details
The implementation uses dataset-specific crop sizes and multi-scale augmentation, while semi-supervised training samples additional unlabeled images under a shared training strategy.
- PASCAL VOC 2012 and Cityscapes use crop sizes of 512 × 512 and 800 × 800, respectively, with scales randomly selected from 0.5 to 1.75.
- Cityscapes uses OHEM loss for supervised training and cross entropy loss for cross pseudo supervision.
- Semi-supervised methods sample 8 additional unlabeled images per iteration, while supervised baselines use batch size 8.
- The method and comparison methods follow the same training strategy and iteration count for fairness.
B. Network Perturbation
CPS relies on two same-architecture networks initialized differently, whose prediction overlap is initially small and increases during later training.
- CPS includes two same-architecture segmentation networks, f(θ1) and f(θ2), initialized differently to create network perturbation.
- The prediction overlap ratio is measured on labeled, unlabeled, and whole datasets during training.
- The overlap ratio is small early in training and increases later, with early diversity helping avoid convergence toward a wrong direction.