PaperlayerView exact revision on arXiv

Source-linked AI summary

Rectifying Pseudo Label Learning via Uncertainty Estimation for Domain Adaptive Semantic Segmentation

Zhedong Zheng, Yi Yang

arXiv:2003.03773v3cs.CV

TL;DR

Unsupervised domain adaptation for semantic segmentation must learn from unlabeled target images despite noisy pseudo labels produced under source–target distribution shift. The paper estimates prediction uncertainty as variance and uses variance regularization to rectify pseudo-label learning, achieving competitive performance across three benchmarks.

  • Problem

    Source-trained models produce inaccurate target pseudo labels under different data distributions, and noisy labels can compromise subsequent adaptation.

  • Method

    The method estimates prediction uncertainty as variance and incorporates it into training through variance regularization as an automatic threshold.

  • Results

    The method achieves significant improvements over conventional pseudo-label learning and competitive performance on two synthetic-to-real and one cross-city benchmark.

  • Takeaways & Limitations

    Uncertainty-aware thresholding rectifies learning from noisy pseudo labels without introducing extra parameters or modules.

Abstract

from arXiv · show

This paper focuses on the unsupervised domain adaptation of transferring the knowledge from the source domain to the target domain in the context of semantic segmentation. Existing approaches usually regard the pseudo label as the ground truth to fully exploit the unlabeled target-domain data. Yet the pseudo labels of the target-domain data are usually predicted by the model trained on the source domain. Thus, the generated labels inevitably contain the incorrect prediction due to the discrepancy between the training domain and the test domain, which could be transferred to the final adapted model and largely compromises the training process. To overcome the problem, this paper proposes to explicitly estimate the prediction uncertainty during training to rectify the pseudo label learning for unsupervised semantic segmentation adaptation. Given the input image, the model outputs the semantic segmentation prediction as well as the uncertainty of the prediction. Specifically, we model the uncertainty via the prediction variance and involve the uncertainty into the optimization objective. To verify the effectiveness of the proposed method, we evaluate the proposed method on two prevalent synthetic-to-real semantic segmentation benchmarks, i.e., GTA5 -> Cityscapes and SYNTHIA -> Cityscapes, as well as one cross-city benchmark, i.e., Cityscapes -> Oxford RobotCar. We demonstrate through extensive experiments that the proposed approach (1) dynamically sets different confidence thresholds according to the prediction variance, (2) rectifies the learning from noisy pseudo labels, and (3) achieves significant improvements over the conventional pseudo label learning and yields competitive performance on all three benchmarks.

1 Introduction

Unsupervised semantic segmentation adaptation addresses source–target distribution differences without requiring costly dense target annotations. This paper estimates prediction uncertainty to automatically threshold and rectify noisy pseudo-label learning, achieving competitive results across three benchmarks.

  • Motivation: Unsupervised adaptation uses labeled source data and unlabeled target data because dense target-domain annotations are costly.The central challenge is the discrepancy between source and target data distributions.
  • Existing approaches: Domain-alignment methods learn shared cross-domain knowledge but can ignore domain-specific information.Existing alignment operates at pixel, feature, or semantic levels.
  • Pseudo-label challenge: Pseudo labels contain noise from source-trained models, while fixed confidence thresholds are difficult to determine across target domains.The noise can compromise subsequent learning.
  • Proposed method: The proposed method models uncertainty as prediction variance and incorporates it through variance regularization compatible with cross-entropy.The regularization provides a pixel-wise automatic threshold for each input image.
  • Contributions: Uncertainty estimation enables automatic thresholding and rectifies learning from noisy pseudo labels without extra parameters or modules.The approach uses uncertainty estimation instead of directly trusting pseudo labels or manually setting confidence thresholds.
  • Results: The method achieves significant improvements over conventional pseudo-label learning and competitive performance on two synthetic-to-real and one cross-city benchmark.The evaluated benchmarks include GTA5 → Cityscapes, SYNTHIA → Cityscapes, and Cityscapes → Oxford RobotCar.

2 Related work

Related work addresses semantic segmentation adaptation through domain alignment, pseudo-label learning, co-training, and uncertainty estimation. The paper positions its approach around annotation uncertainty for learning from noisy pseudo labels.

  • Domain alignment: Domain-alignment methods transfer visual style or align features and semantic outputs to learn shared source–target knowledge.These methods modify appearance or align representations at different abstraction levels.
  • Pseudo-label learning: Pseudo-label methods predict labels for unlabeled target data and fine-tune the model as in supervised learning.Prior work applies pseudo labels to semantic segmentation adaptation and target-domain distribution learning.
  • Problem setting: Source-trained models generate inaccurate target pseudo labels because source and target data distributions differ.This noise motivates methods that avoid treating every pseudo label as reliable ground truth.
  • Co-training: Co-training uses two classifiers to learn complementary information, while related adaptation methods introduce discrepancy-based losses.The proposed method instead leverages discrepancy between classifiers naturally through two classifiers on different intermediate layers.
  • Uncertainty estimation: Uncertainty estimation can target input data, annotations, or model weights; this paper focuses on annotation uncertainty.Its goal is to predict whether an annotation is correct and learn from noisy pseudo labels.

3 Methodology

The method revisits pseudo-label adaptation, where source-trained predictions can be noisy, and introduces uncertainty estimation through prediction variance to rectify learning. A two-classifier model supplies variance estimates, which enter a variance-regularized objective that automatically adjusts pixel-wise trust in pseudo labels.

  • 3.1 Problem Definition: Unsupervised adaptation estimates target-domain parameters from labeled source data and unlabeled target inputs, whose ground-truth labels remain unknown during training.
  • 3.2 Pseudo Label Learning Revisit: Pseudo-label learning generates target labels with a source-trained model, then minimizes prediction bias by treating fixed pseudo labels as true labels.The conventional objective commonly uses cross-entropy loss.
  • 3.2 Pseudo Label Learning Revisit: Source-target distribution differences make pseudo labels inherently inaccurate, allowing incorrect annotations to compromise subsequent adaptation.
  • 3.3 Uncertainty Estimation: Prediction variance is approximated from the discrepancy between primary and auxiliary classifier outputs, independently of the pseudo label.The auxiliary classifier uses a shallower layer, while the primary classifier uses a deeper layer; their prediction discrepancy estimates model uncertainty.
  • 3.4 Variance Regularization: Variance regularization combines uncertainty with the conventional objective, reducing pressure to minimize prediction bias when variance is high while penalizing persistently high variance.The reciprocal variance term is replaced by exp(−Var) to stabilize training when variance can be zero.
  • 3.4 Variance Regularization: The rectified loss provides pixel-wise thresholds: coherent predictions reinforce pseudo-label trust, whereas ambiguous high-variance areas are neglected.The training cost approximately equals conventional pseudo-label learning because no extra modules are introduced.

4 Experiment

Experiments across synthetic-to-real and cross-city benchmarks show that uncertainty-aware pseudo-label learning improves adaptation over conventional pseudo-label learning, while variance regularization provides dynamic pixel-wise thresholding and avoids fitting all noisy labels.

  • Comparisons with state-of-the-art methods: 50.3% mIoU on GTA5 →Cityscapes exceeds MRNet+Pseudo by +2.0% and MRNet by 4.8 mIoU.The proposed method also surpasses PatchAlign, CBST, and MRKLD without using prior location knowledge.
  • Comparisons with state-of-the-art methods: 47.9 mIoU for 16 categories and 54.9 mIoU* for 13 categories on SYNTHIA →Cityscapes improve over MRNet+Pseudo by +1.4% and +1.1%, respectively.The proposed method also exceeds APODA by 1.8% mIoU* in the reported comparison.
  • Comparisons with state-of-the-art methods: 74.4% mIoU on Cityscapes →Oxford RobotCar is only +0.5% above MRNet+Pseudo, although the method achieves the best IoU on 6 of 9 classes.The traffic-sign class improves by +5.7%, while the baseline remains competitive on this benchmark.
  • Further Evaluations: Variance regularization acts as a dynamic threshold that supplies different thresholds for different pixels, outperforming handcrafted confidence thresholds on GTA5 →Cityscapes.Table 5 reports mIoU accuracy for conventional thresholds and the proposed regularization.
  • Further Evaluations: With noisier pseudo labels from AdaptSegNet, the proposed method improves mIoU from 42.4% to 47.4%, exceeding conventional pseudo learning at 46.8% mIoU.The result tests scalability beyond pseudo labels generated by MRNet.
  • Further Evaluations: The proposed method converges to a non-zero loss and variance regularization penalizes uncertain wrong predictions, while conventional pseudo-label learning converges to zero by fitting all pseudo labels.Without dropout, the method still reaches 49.6% mIoU; suitable dropout rates generally produce results around 50% mIoU.

5 Conclusion

The paper identifies noisy pseudo-label learning as a challenge in adaptive semantic segmentation and proposes variance regularization to estimate uncertainty and rectify training. Without extra parameters or modules, the method achieves competitive performance on three benchmarks.

  • The method estimates prediction uncertainty during training and incorporates it into optimization through variance regularization.
  • Variance regularization helps the model learn from noisy pseudo labels without introducing extra parameters or modules.
  • The proposed approach achieves competitive performance across three semantic segmentation adaptation benchmarks.
Loading 2003.03773v3…