PaperlayerView exact revision on arXiv

Source-linked AI summary

Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels

Yuchao Wang, Haochen Wang, Yujun Shen, Jingjing Fei, Wei Li, Guoqiang Jin, Liwei Wu, Rui Zhao, Xinyi Le

arXiv:2203.03884v2cs.CV

TL;DR

Semi-supervised semantic segmentation must use unlabeled pixels effectively, yet confidence filtering leaves unreliable predictions unused. U2PL separates pixels by entropy, uses reliable predictions positively, and queues unreliable pixels as category-specific negatives; it outperforms state-of-the-art methods across benchmarks and settings, with improved ambiguous-region performance.

  • Problem

    Confidence filtering avoids incorrect pseudo-labels but leaves many ambiguous pixels unused, limiting how unlabeled data contribute to semi-supervised segmentation.

  • Method

    U2PL separates reliable and unreliable pixels by entropy, uses reliable predictions as positive pseudo-labels, and stores unreliable pixels in category-wise negative-sample queues.

  • Results

    U2PL surpasses state-of-the-art competitors on PASCAL VOC 2012 and Cityscapes across varied training settings, with mIoU of 79.30% under the 1/4 partition protocol.

  • Takeaways & Limitations

    Using unreliable pseudo-labels provides useful negative information and is associated with better performance in ambiguous regions such as object borders.

  • Takeaways & Limitations

    Training U2PL is time-consuming compared with fully supervised methods, reflecting a common semi-supervised-learning disadvantage.

Abstract

from arXiv · show

The crux of semi-supervised semantic segmentation is to assign adequate pseudo-labels to the pixels of unlabeled images. A common practice is to select the highly confident predictions as the pseudo ground-truth, but it leads to a problem that most pixels may be left unused due to their unreliability. We argue that every pixel matters to the model training, even its prediction is ambiguous. Intuitively, an unreliable prediction may get confused among the top classes (i.e., those with the highest probabilities), however, it should be confident about the pixel not belonging to the remaining classes. Hence, such a pixel can be convincingly treated as a negative sample to those most unlikely categories. Based on this insight, we develop an effective pipeline to make sufficient use of unlabeled data. Concretely, we separate reliable and unreliable pixels via the entropy of predictions, push each unreliable pixel to a category-wise queue that consists of negative samples, and manage to train the model with all candidate pixels. Considering the training evolution, where the prediction becomes more and more accurate, we adaptively adjust the threshold for the reliable-unreliable partition. Experimental results on various benchmarks and training settings demonstrate the superiority of our approach over the state-of-the-art alternatives.

1. Introduction

Semi-supervised semantic segmentation seeks to exploit abundant unlabeled data because annotated data are costly, but confidence filtering leaves ambiguous pixels unused. U2PL instead treats unreliable predictions as category-specific negative evidence and reports stronger performance across benchmarks and settings.

  • Semi-supervised segmentation uses limited labeled samples and numerous unlabeled images, making effective unlabeled-data utilization critical.
  • Confidence-based pseudo-label filtering reduces confirmation bias but leaves ambiguous pixels unused throughout training.
  • Unlearned classes can produce insufficient and categorically imbalanced training when accurate pseudo-labels are difficult to assign.The paper illustrates this issue with the chair category.
  • U2PL separates pixels by prediction entropy, uses reliable predictions as positive pseudo-labels, and stores unreliable pixels as negative samples in category-wise queues.The framework is designed to use every pixel while avoiding direct supervision from unreliable class assignments.
  • U2PL surpasses state-of-the-art competitors on PASCAL VOC 2012 and Cityscapes across a wide range of training settings.Visualizations particularly show better performance in ambiguous regions such as object borders.

2. Related Work

Prior semi-supervised segmentation methods commonly use consistency, entropy minimization, augmentation, or confidence-filtered self-training. U2PL focuses on unreliable pixels and uses their certainty about unlikely categories as negative learning information.

  • Semi-supervised learning methods commonly follow consistency regularization, entropy minimization, or self-training paradigms.
  • Augmentation-based self-training methods use CutOut, CutMix, or ClassMix but pay limited attention to semantic-segmentation-specific characteristics.
  • Confidence thresholds and uncertainty-aware methods typically filter unreliable pixels, which U2PL instead includes during training.
  • U2PL distinguishes unlikely categories for unreliable pixels, contrasting with negative-learning methods that select high-confidence negative samples.

3. Method

U2PL uses entropy to separate reliable pseudo-labels from unreliable pixels, then incorporates both through supervised, unsupervised, and contrastive objectives. Unreliable pixels become category-specific negative samples stored in FIFO memory banks, while thresholds adapt during training.

  • Overview: U2PL follows a teacher–student self-training framework in which the teacher is EMA-updated from the student and produces predictions for unlabeled images.Labeled images train the student directly; both models use an encoder, segmentation head, and representation head.
  • Optimization: The total objective combines supervised loss, unsupervised cross-entropy on reliable pseudo-labels, and pixel-level InfoNCE contrastive loss for unreliable pseudo-labels.The loss weights include λu for the unsupervised term and λc for the contrastive term.
  • Pseudo-Labeling: Entropy partitions teacher predictions into reliable pixels used as pseudo-labels and unreliable pixels excluded from direct supervision.The entropy threshold is defined by a training-time quantile, and unreliable-pixel proportions are dynamically adjusted as training progresses.
  • Unreliable Pseudo-Labels: Unreliable pixels are treated as negative samples for classes they probably do not belong to, using entropy and category-rank conditions to select candidates.For unlabeled pixels, candidates must be unreliable, unlikely to belong to the target class, and outside the most unlikely classes.
  • Contrastive Sampling: Contrastive sampling uses class-specific anchor pixels, class-center positive samples, and negative candidates to improve representation discrimination while limiting computation.Qualified unlabeled anchors are reliable according to the entropy threshold, whereas unreliable pixels supply negative candidates.
  • Memory Banks: Category-wise FIFO memory banks maintain negative samples when mini-batches contain too few candidates for particular categories.The training algorithm pushes negatives into each class queue and removes the oldest entries when necessary.

4. Experiments

U2PL is evaluated on PASCAL VOC 2012 and Cityscapes across partition protocols against recent semi-supervised segmentation methods. It consistently improves segmentation, including ambiguous regions, and ablations support using unreliable pixels with adaptive partitioning and rank thresholds.

  • 4.1. Setup: U2PL is evaluated on PASCAL VOC 2012 and Cityscapes under 1/2, 1/4, 1/8, and 1/16 partition protocols using a ResNet-101 backbone with DeepLabv3+ decoder.The comparisons use common network architectures and recent semi-supervised segmentation baselines.
  • 4.2. Comparison with Existing Alternatives: +22.21%, +14.23%, +7.78% and +4.47% over the supervised baseline under 1/16, 1/8, 1/4 and 1/2 protocols, respectively, on classic PASCAL VOC 2012.U2PL also outperforms PC2Seg under all four partition protocols.
  • 4.3. Ablation Studies: Unreliable pseudo-pixels outperform reliable, all-pixel, and other candidate selections in the PASCAL VOC 2012 ablation.Unreliable candidates are selected from the top 20% highest-entropy pixels, while reliable candidates come from the bottom 20%.
  • 4.3. Ablation Studies: PRT settings r_l = 3 and r_h = 20 outperform other tested options by a large margin.The study reports that lower or higher rank thresholds can respectively retain false negatives or reduce semantic relevance.
  • 4.3. Ablation Studies: Adding L_c improves the baseline by +4.06%, while the full component combination reaches +6.05% or +5.47% depending on the included strategy.The reported full method achieves 79.30% mIoU; α_0 = 20% is the best initial reliable-unreliable partition.
  • 4.2. Comparison with Existing Alternatives: 79.30% mIoU is achieved under the blender 1/4 partition protocol after combining the proposed components.The blender protocol uses 2,646 labeled and 7,396 unlabeled images in the qualitative comparison setting.
  • 4.4. Qualitative Results: U2PL produces better segmentation on ambiguous regions, including borders between different objects, than the compared methods.Unfiltered contrastive learning can harm performance and sometimes perform worse than training with labeled data only.

5. Conclusion

U2PL incorporates unreliable pseudo-labels into semi-supervised segmentation and outperforms state-of-the-art methods, with especially strong qualitative results on ambiguous regions. The method remains more time-consuming than fully supervised training.

  • U2PL includes unreliable pseudo-labels during training and outperforms many state-of-the-art methods.
  • Qualitative results show especially better performance on borders between semantic objects and other ambiguous regions.
  • Training U2PL is time-consuming compared with fully supervised methods, reflecting a common semi-supervised learning disadvantage.The paper identifies training optimization as an area for future exploration.

A. More Details for Reproducibility

The appendix specifies augmentation, optimization, memory-queue, and hyper-parameter settings used to support reproducibility across PASCAL VOC 2012 and Cityscapes experiments.

  • Optimization: SGD is used throughout, with dataset-specific learning rates and weight decay: 0.001 and 0.0001 for PASCAL VOC, versus 0.01 and 0.0005 for Cityscapes.The temperature τ is set to 0.5 for both datasets.
  • Data augmentation: Experiments use random cropping, horizontal flipping, and random scaling from 0.5 to 2.0 for both datasets.These augmentations are applied to prevent overfitting, following previous methods.
  • Evaluation protocol: The Cityscapes 1/2 partition uses 1,488 labeled and 1,487 unlabeled images.Figure A1 reports qualitative results under this partition protocol.
  • Memory queue and baselines: The category-specific memory queue contains 50,000 background samples and 30,000 samples for each foreground category.The baselines SupOnly, MT, and CutMix are re-implemented, with CutMix differing from MT through unlabeled-image CutMix augmentation.
  • Hyper-parameters: The appendix identifies M, N, and δp as contrastive-learning parameters, λc, η, and τ as training parameters, and α0, rl, and rh as U2PL parameters.The hyper-parameters are summarized in Table A1.

B. More Results on Cityscapes

Cityscapes ablations show that selecting unreliable pixels is more effective than selecting reliable or entropy-agnostic pixels, while qualitative comparisons show U2PL improves predictions and avoids harm from unfiltered contrastive learning.

  • Quantitative results: Unreliable pixel selection outperforms the other reliability options in Cityscapes mIoU results.Table A2 defines unreliable pixels as those in the top 20% of entropy scores, reliable pixels as the bottom 20%, and all as entropy-agnostic sampling.
  • Qualitative results: U2PL outperforms other methods qualitatively on the Cityscapes validation set by using unreliable pseudo-labels.Figure A1 compares labeled-only training, vanilla contrastive learning, and U2PL.
  • Qualitative results: Unfiltered contrastive learning can harm performance, producing worse results than labeled-only training in some examples.The cited examples are the first and fourth rows of Figure A1.
  • Qualitative results: The visual comparisons are presented as evidence that U2PL strengthens the reliability of unreliable prediction labels.This claim is made in the discussion of Figure A1.

C. Alternative of Contrastive Learning

U2PL can use unreliable pseudo-labels through binary classification as well as contrastive learning. On Cityscapes, its binary-cross-entropy variant improves over supervised-only training across all reported partial protocols.

  • Binary-classification alternative: U2PL is not limited to contrastive learning because binary cross-entropy can also use unreliable pseudo-labels.For an anchor z^c_i belonging to class c, the method computes the BCE loss using negative samples.
  • Binary-classification alternative: The binary-classification formulation uses C total classes, M anchor pixels, N negative samples, cosine similarity, and temperature τ.These quantities define the terms used in the BCE formulation.
  • Results: On Cityscapes, the BCE variant improves over supervised-only training by +3.77%, +0.40%, +1.48%, and +0.53% under 1/16, 1/8, 1/4, and 1/2 protocols.These gains are reported from Tables A3 and A4 discussion.

D.1. More Hyper-parameters on VOC

The VOC hyper-parameter studies examine base learning rate and temperature under the reported partition protocol. U2PL performs best with a base learning rate of 0.001 and temperature τ = 0.5.

  • Ablation Coverage: The appendix includes ablation tables for VOC partition protocols and hyper-parameter studies.The supplied captions identify tables covering binary-classification partition protocols and temperature or learning-rate ablations.
  • Temperature: τ = 0.5 achieves the best U2PL results among the tested temperature values.Temperature adjusts the importance assigned to hard samples, while values that are too large or too small adversely affect overall performance.

D.2. Ablation Studies on Cityscapes

Cityscapes ablations show that the balance between reliable and unreliable pixels and the initial partition proportion strongly affect performance. Feature visualizations further associate U2PL with clearer decision boundaries than supervised-only training.

  • Probability Rank Threshold: r_l = 3 and r_h = 20 outperform other probability-rank-threshold options by a large margin.This ablation verifies that the selected balance between rank thresholds promotes performance.
  • Ablation Coverage: The appendix captions identify Cityscapes ablations for probability-rank thresholds and initial partition proportions.These studies are reported on the Cityscapes validation set.
  • Feature Visualization: U2PL produces clearer feature decision boundaries than the supervised-only method in the t-SNE visualization.The visualization is presented as an explanation for why U2PL works from a feature perspective.
Loading 2203.03884v2…