Source-linked AI summary
Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentation
Pan Zhang, Bo Zhang, Ting Zhang, Dong Chen, Yong Wang, Fang Wen
TL;DR
Unsupervised domain-adaptive segmentation must learn from noisy target pseudo labels while handling dispersed target features caused by domain discrepancy. ProDA uses prototypes and feature distances for online label correction, aligns prototypical assignments across target views, and distills knowledge to a self-supervised model. It reports substantial gains over state-of-the-art methods, including 57.5 and 55.5 Cityscapes mIOU for GTA5 and SYNTHIA adaptation, respectively.
Problem
Self-training for unsupervised domain-adaptive segmentation suffers from noisy pseudo labels and dispersed target features caused by source-target domain discrepancy.
Method
ProDA uses relative feature distances to class prototypes for online pseudo-label denoising, aligns soft prototypical assignments across target views, and distills knowledge to a self-supervised pretrained model.
Results
57.5 and 55.5 Cityscapes mIOU are reported for adaptation from GTA5 and SYNTHIA, respectively, with ProDA substantially outperforming state-of-the-art methods.
Takeaways & Limitations
Prototype-based denoising and target-structure learning improve self-training, while distillation to a self-supervised pretrained model further boosts performance.
Takeaways & Limitations
Prototypical consistent learning may suffer degeneration when one cluster becomes empty, requiring a regularization term to amend this issue.
Abstract
from arXiv · showhide
Self-training is a competitive approach in domain adaptive segmentation, which trains the network with the pseudo labels on the target domain. However inevitably, the pseudo labels are noisy and the target features are dispersed due to the discrepancy between source and target domains. In this paper, we rely on representative prototypes, the feature centroids of classes, to address the two issues for unsupervised domain adaptation. In particular, we take one step further and exploit the feature distances from prototypes that provide richer information than mere prototypes. Specifically, we use it to estimate the likelihood of pseudo labels to facilitate online correction in the course of training. Meanwhile, we align the prototypical assignments based on relative feature distances for two different views of the same target, producing a more compact target feature space. Moreover, we find that distilling the already learned knowledge to a self-supervised pretrained model further boosts the performance. Our method shows tremendous performance advantage over state-of-the-art methods. We will make the code publicly available.
1. Introduction
Unsupervised domain adaptation for semantic segmentation transfers knowledge from labeled synthetic source images to unlabeled real target images, but domain misalignment and noisy self-training pseudo labels limit performance. ProDA addresses these issues with online prototype-based denoising, compact target-structure learning, and distillation to a self-supervised pretrained model.
- Unsupervised domain adaptation transfers knowledge from labeled synthetic images to real images without accessing target-domain labels.
- Self-training iteratively retrains the network on confident target pseudo labels, but its performance still lags behind supervised learning.
- Strict confidence thresholds cannot ensure pseudo-label correctness, while domain gaps produce dispersed target features that hinder reliable learning.
- ProDA estimates class-wise pseudo-label likelihoods from relative feature distances to on-the-fly class prototypes, progressively correcting labels during training.
- ProDA aligns soft prototypical assignments across differently augmented views to produce a more compact target feature space.
- 57.5 mIOU on Cityscapes after GTA5 adaptation and 55.5 after SYNTHIA adaptation demonstrate ProDA's reported advantage over prior works.
2. Related Work
Related work approaches unsupervised domain adaptation through distribution alignment, semi-supervised learning, self-training, unsupervised representation learning, and noisy-label correction. These studies motivate ProDA's prototype-based online correction and self-supervised knowledge distillation.
- Unsupervised domain adaptation: Domain alignment methods reduce source-target distribution mismatch at image, feature, or output levels, but global alignment does not guarantee low target error.
- Unsupervised domain adaptation: Entropy minimization sharpens predictions on unlabeled targets but can make networks over-confident in false predictions, motivating self-training with pseudo labels.
- Unsupervised representation learning: Unsupervised representation learning uses pretext tasks and contrastive learning, with prominent methods substantially closing the supervised-learning gap.
- Learning from noisy labels: Noisy-label research includes robust losses, agreement among multiple learners, online prototype correction, and knowledge distillation for transferring cleaner knowledge.
- Unsupervised domain adaptation: Figure 1 depicts self-training errors caused by target-distribution-unaware decision boundaries and dispersed target features that linear classifiers hardly differentiate.
3. Preliminary
The preliminary setup considers a segmentation network trained on labeled source data and adapted to an unlabeled target dataset sharing the same classes. Conventional self-training converts source-network predictions into thresholded pseudo labels and updates them only between training stages.
- The target dataset has no accessible ground-truth labels, while source and target domains share K semantic classes.
- The segmentation network is modeled as a feature extractor f followed by a classifier g.
- A source-trained network may generalize poorly to target data because of the domain gap, so self-training optimizes cross-entropy using target pseudo labels.
- For each target pixel, the pseudo label is typically the most probable class predicted by the source network.
- Only predictions exceeding a confidence threshold contribute to retraining, and pseudo labels are updated until convergence before the next training stage.
4. Method
The method jointly denoises target pseudo labels online and learns a compact target feature structure using prototype-based assignments, then transfers the learned knowledge to a self-supervised pretrained student.
- Online pseudo-label denoising: The method fixes soft pseudo labels and progressively reweights them with class-wise probabilities updated from newly learned knowledge.This online strategy is designed to avoid both delayed correction and trivial solutions from simultaneous unrestricted updates.
- Online pseudo-label denoising: Prototype-distance modulation downweights a class probability when the target feature lies far from that class prototype, approximating class-specific trust.The modulation weight is computed using a softmax over feature distances, with momentum-encoder features used for more reliable estimation.
- Prototype computation: Prototypes are initialized from predicted target pseudo labels and updated as moving averages of mini-batch cluster centroids during training.This on-the-fly estimation avoids the computational cost of recalculating prototypes over the full target set.
- Target structure learning: The method aligns prototypical assignments between weak and strong views using KL divergence, encouraging consistent labels for adjacent feature points and compact target features.A momentum encoder supplies the more reliable assignment used to guide the trainable encoder.
- Target structure learning: Because prototypical consistency can leave a cluster empty, the method adds a regularizer encouraging outputs to distribute evenly across classes.The complete adaptation objective combines online label correction, prototypical consistency, and this regularization term.
- Distillation to a self-supervised model: After adaptation converges, knowledge is distilled from the learned target model into a same-architecture student initialized with self-supervised pretrained weights.The student mimics the teacher on unlabeled target images, and the distillation can be repeated after convergence.
5. Experiments
Experiments evaluate ProDA on synthetic-to-real semantic segmentation, compare it with prior methods, and analyze its components, pseudo-label refinement, feature structure, and robustness. ProDA achieves strong adaptation results while online denoising, structure learning, and self-supervised initialization each contribute to performance.
- Experimental setup: Experiments use DeepLabv2 with a ResNet-101 backbone and evaluate GTA5→Cityscapes and SYNTHIA→Cityscapes adaptation.GTA5 and SYNTHIA provide synthetic source images, while Cityscapes supplies the real target domain and shared category protocols.
- Comparisons with state-of-the-art methods: ProDA achieves 57.5 mIoU on GTA5→Cityscapes, outperforming existing methods and improving over the non-adapted baseline by 20.9.It obtains the best scores in 15 of 19 categories, with gains concentrated in challenging small or rare objects.
- Comparisons with state-of-the-art methods: ProDA reaches 55.5 and 62.0 mIoU on SYNTHIA→Cityscapes across 16 and 13 categories, respectively.It achieves the best result in 11 of 16 categories and improves over the nonadaptive model by 20.6.
- Ablation and component analysis: Online denoising raises mIoU to 52.3, exceeding the prior best score by 2.0 in the ablation study.The source model scores 36.6, warm-up adds 5.0, vanilla self-training reaches 45.2, and symmetric cross-entropy adds 0.4.
- Pseudo-label refinement: ProDA’s pseudo labels improve throughout training, identifying tiny objects progressively and correcting most incorrect labels by 40k iterations.Unlike conventional self-training’s step-wise updates, ProDA quickly attains high-quality pseudo labels during training.
- Ablation and component analysis: Fixed soft labels prevent degenerate solutions during simultaneous feature learning and label updating, while prototypical reweighting improves mIoU by more than 5.0.Hard labels provide an additional improvement of approximately 0.2 over soft labels.
- Ablation and component analysis: Target structure learning improves performance from 45.6 to 47.6 without label denoising, assists denoising through compact clusters, and adds a 1.4 gain.Initialization from a SimCLRv2 self-supervised model further improves mIoU by 0.6 compared with resuming the previous stage.
- Robustness analysis: ProDA is not sensitive to the pseudo-label threshold or prototype momentum across the tested ranges.The method therefore omits thresholding for convenient usage.
6. Conclusions
ProDA uses prototypes to denoise target pseudo labels and compact the target feature space, while distillation to a self-supervised pretrained model further improves adaptation.
- ProDA uses prototypes to online denoise pseudo labels and learn a compact target-domain feature space.The method addresses noisy pseudo labels and dispersed target features through prototype-based learning.
- Knowledge distillation to a self-supervised pretrained model further boosts ProDA's performance.
- ProDA outperforms state-of-the-art methods by a large margin and greatly reduces the gap with supervised learning.
Appendix A. Influences of Design Choices
The appendix studies prototype initialization, target-view augmentation, denoising temperature, loss coefficients, and loss weights to characterize ProDA's design choices.
- Prototype initialization strategy: Target prototypes can be initialized from target pseudo predictions or source-domain ground-truth labels, trading pseudo-label noise against domain gap.
- Strong augmentation: 52.7 mIoU results from random-crop augmentation alone, while RandAugment and CutOut improve mIoU by 0.78 and 0.5, respectively.The strongest augmentation gives the best performance.
- Temperature during prototypical denoising: τ = 1 is used for prototypical denoising, balancing nearest-prototype dominance against equal weighting of all prototypes.
- Symmetric cross-entropy loss: The final result is not sensitive to SCE coefficients when β is not too small, using α = 0.1 and β = 1.
- Loss weights: The KL and regularization loss weights are evaluated in the first training stage on GTA5 →Cityscapes.
Appendix B. Algorithm
ProDA training comprises a warmup and prototype-based target adaptation, followed by knowledge distillation to a self-supervised model using source and target data.
- Prototype-based adaptation: The first stage updates pseudo labels and calculates soft labels for weak and strong target views during prototypical denoising and target structure learning.
- Knowledge distillation: The second and third stages apply knowledge distillation to a self-supervised model, including student-model initialization and probability calculation.
- Inputs: The algorithm takes source images and labels, target images, prototype momentum, weak and strong augmentations, a pretrained SimCLRv2 model, and a pseudo-label threshold as inputs.
- Warmup: A warmup first trains the segmentation model using source and target datasets according to the specified procedure.
- Training loop: The procedure repeatedly obtains source and target images and tunes the model during training.
Appendix C. Detailed Ablation study
The detailed ablation study covers all 19 GTA5 →Cityscapes classes and examines the contributions of self-training, prototypical denoising, structure learning, and knowledge distillation.
- Scope: The ablation study reports results for all 19 classes on GTA5 →Cityscapes.
- Components: The component ablation distinguishes self-training, prototypical denoising, and structure learning, while knowledge distillation can be applied in the last two training stages.
Appendix D. Qualitative comparison
The appendix presents qualitative semantic-segmentation results on Cityscapes and qualitative comparisons among different methods.
- Figure 6 compares input images, results before adaptation, conventional self-training, and ProDA on Cityscapes.
- Figure 7 presents qualitative comparisons of different methods.
- Figure 8 presents qualitative comparisons of different methods.