PaperlayerView exact revision on arXiv

Source-linked AI summary

Re-distributing Biased Pseudo Labels for Semi-supervised Semantic Segmentation: A Baseline Investigation

Ruifei He, Jihan Yang, Xiaojuan Qi

arXiv:2107.11279v2cs.CV

TL;DR

Self-training for semi-supervised semantic segmentation is vulnerable to pseudo-label bias from long-tailed class distributions. The paper introduces DARS to align pseudo labels with labeled-data distributions and adds progressive augmentation and labeling; experiments on Cityscapes and PASCAL VOC 2012 show favorable performance against state-of-the-art methods.

  • Problem

    Long-tailed class distributions bias self-training pseudo labels toward majority classes, producing a mismatch with true distributions and harming effective learning for tail categories.

  • Method

    DARS uses distribution alignment, classwise thresholding, and random sampling to match pseudo-label distributions to labeled-data distributions, alongside progressive augmentation and labeling.

  • Results

    8.89% mIoU is reported as the Cityscapes gain in the 1 8 split setting, while PASCAL VOC 2012 performance exceeds previous state-of-the-art by 4.49% mIoU.

  • Takeaways & Limitations

    The method is presented as a simple, generic, and efficient baseline that can be incorporated into other self-training pipelines.

  • Takeaways & Limitations

    Confidence overlap caused by over-confident predictions complicates threshold-based distribution matching, and systematic errors on small objects limit self-training performance there.

Abstract

from arXiv · show

While self-training has advanced semi-supervised semantic segmentation, it severely suffers from the long-tailed class distribution on real-world semantic segmentation datasets that make the pseudo-labeled data bias toward majority classes. In this paper, we present a simple and yet effective Distribution Alignment and Random Sampling (DARS) method to produce unbiased pseudo labels that match the true class distribution estimated from the labeled data. Besides, we also contribute a progressive data augmentation and labeling strategy to facilitate model training with pseudo-labeled data. Experiments on both Cityscapes and PASCAL VOC 2012 datasets demonstrate the effectiveness of our approach. Albeit simple, our method performs favorably in comparison with state-of-the-art approaches. Code will be available at https://github.com/CVMI-Lab/DARS.

1. Introduction

Semi-supervised self-training reduces annotation demands but produces pseudo-label bias under long-tailed class distributions. The paper proposes DARS and progressive training strategies, reporting gains on Cityscapes and PASCAL VOC 2012.

  • Motivation: Long-tailed segmentation data biases self-training pseudo labels toward dominant categories, creating a mismatch with true class distributions.This mismatch can hinder further progress and harm learning for tail categories.
  • Contributions: DARS aligns pseudo-label distributions with the true class distribution estimated from labeled data.It combines classwise thresholding with random sampling to address confidence overlap.
  • Contributions: Progressive augmentation and labeling gradually strengthen augmentation and increase the labeling ratio during self-training.The strategy limits early exposure to noisy or difficult examples while reducing overfitting to easy pseudo-labeled examples.
  • Results: 8.89% mIoU is the reported Cityscapes gain in the 1 8 split setting, approaching fully supervised results.The paper also reports a 4.49% mIoU improvement over previous state-of-the-art on PASCAL VOC 2012.
  • Additional analysis: Performance gains from additional unlabeled data gradually saturate in the high-data regime.The paper analyzes potential bottlenecks and suggests future research directions.

2. Related Work

Related work covers supervised segmentation architectures, consistency-based and self-training semi-supervised learning, and prior semi-supervised segmentation methods. The paper positions its approach most closely against self-training methods that do not correct pseudo-label bias.

  • Supervised Semantic Segmentation: Supervised segmentation advances include multi-scale inputs, pyramid spatial pooling, dilated convolutions, and encoder-decoder architectures.The paper uses PSPNet in its main experiments and Deeplabv2 for comparison with prior work.
  • Semi-Supervised Learning: Semi-supervised learning research includes consistency training and self-training based on pseudo labeling or entropy minimization.Consistency methods enforce prediction stability under perturbations, whereas self-training uses model predictions as pseudo labels.
  • Semi-Supervised Semantic Segmentation: Prior semi-supervised segmentation methods use adversarial learning, correction models, consistency constraints, and related mechanisms to select or refine predictions.These approaches target reliable pseudo labels or prediction consistency in segmentation.
  • Semi-Supervised Semantic Segmentation: The paper is most closely related to self-training methods that use confidence thresholds or class sample limits without exploiting pseudo-label distribution bias.It contrasts these methods with its focus on correcting the bias in pseudo labeling.

3. Method

The method alternates teacher-generated pseudo-labeling with student training, using DARS to align pseudo-label class frequencies with labeled-data frequencies and progressive augmentation and labeling to support iterative improvement.

  • Overview: Self-training alternates generating pseudo labels from a teacher model with training a student model on labeled and pseudo-labeled data.The student resumes from the teacher, and the process repeats until no further performance gain is achieved.
  • Unbiased Pseudo Label Generation: DARS targets distribution mismatch by using the labeled-data class distribution as the true distribution for pseudo-label generation.The method is designed to produce pseudo-labeled data whose class distribution matches the estimated true distribution.
  • Unbiased Pseudo Label Generation: Category-specific confidence thresholds select pixels while controlling the labeling ratio and encouraging higher-confidence pixels to receive pseudo labels.Pixels below their category threshold receive an ignore label and do not contribute to training.
  • Unbiased Pseudo Label Generation: Confidence overlapping makes thresholding insufficient, because many over-confident pixels—especially in head categories—share indistinguishable confidence values.For Road in Cityscapes, 81.3% of pixels have confidence 1, causing threshold-based counts to exceed desirable class totals.
  • Unbiased Pseudo Label Generation: Random sampling addresses excess pixels from confidence overlap, while distribution alignment primarily helps categories without severe overlap.The method uses random sampling to obtain desirable pixel counts and enlarge the spatial coverage of centralized high-confidence pixels.
  • Progressive Data Augmentation and Labeling: Progressively increasing labeling ratios and augmentation strength introduces new information during iterative training.Stronger augmentation creates unseen or harder cases, while the progressive schedule starts weak and expands random-scaling ranges over stages.

4. Experiments

Experiments evaluate DARS across Cityscapes and VOC12, using multiple backbones, labeled-data splits, ablations, and additional analyses. DARS improves overall and tail-class performance, while progressive training and data augmentation provide further gains.

  • Experimental Setup: Cityscapes experiments use 1/8 and 1/4 labeled splits, with PSPNet-ResNet50 for the main evaluation and Deeplabv2 for fair comparison.The study reports validation results, iterative training, and single-scale testing; PSPNet is selected as a trade-off among reproducibility, performance, and cost.
  • VOC12: 4.49% mIoU is DARS’s improvement over CCT on VOC12 using the PSPNet50 backbone.The comparison also reports that DARS outperforms other previous methods.
  • Ablation Studies: 68.01% mIoU is achieved by DARS in a single self-training round, exceeding ST by 2.31% and CBST by 1.72%.DARS remains 0.7% above DA+TS, indicating that temperature scaling helps but is sub-optimal for confidence overlapping and distribution alignment.
  • Ablation Studies: DARS improves tail-class performance, ranking in the top two for 10 of 13 tail classes and exceeding CBST by 2.9% Tail mIoU.It also improves truck performance by over 8.4% mIoU and is reported to prevent collapse toward head classes through pseudo-label distribution alignment.
  • Progressive Strategy: 1.63% gains result when round-2 training combines a 50% labeled ratio with stronger augmentation, compared with round-1 results.Directly applying stronger augmentation in round 1 can instead degrade performance, supporting progressive introduction of harder examples.
  • Additional Results and Analysis: Held-out test sets show similar gains to validation sets, while additional unlabeled data improves performance until saturation.The analysis attributes high-data bottlenecks partly to saturation for originally rare classes and systematic network errors on small objects, which reduce pseudo-label quality.

5. Conclusion

The paper presents DARS to calibrate pseudo-label bias, paired with progressive augmentation and labeling for iterative self-training. Experiments on Cityscapes and VOC12 show that the simple method can outperform existing sophisticated approaches.

  • DARS calibrates pseudo-label bias, while progressive data augmentation and labeling support iterative self-training.
  • Experiments on Cityscapes and VOC12 demonstrate that the combined approach can outperform existing sophisticated approaches.

Outline

The supplementary material adds Cityscapes analyses of labeling ratios, augmentation magnitude, pseudo-label visualizations, and qualitative method comparisons, alongside additional semi-supervised results.

  • A.1. Parameter Analysis: The supplement analyzes labeling ratio and data augmentation magnitude, and explains the progressive augmentation strategy.
  • A.3–A.4. Visualizations and qualitative results: It also provides pseudo-label visualizations and qualitative comparisons among different methods.
  • Additional experimental results for semi-supervised s... are included.

A.1. Parameter Analysis

The parameter analysis shows that progressively enlarging the labeling ratio and random-scaling range improves performance across relatively broad settings, supporting the robustness of the progressive strategy.

  • Labeling Ratio: Without enlarging the labeling ratio, iterative training improves performance from 68.01% to 68.27%; with enlargement, the largest gain reaches 68.93% at α=50%.
  • Labeling Ratio: Noticeable performance boosts occur across a labeling-ratio range of 40%∼60%.
  • Data Augmentation Magnitude: The progressive strategy increases random-scaling magnitude by lowering the 0.25 bound by βmin and raising the 1.0 bound by βmax.
  • Data Augmentation Magnitude: 68.97%→69.64% is obtained by appropriately enlarging the random-scaling range, with the best result at βmin = 0.2 and βmax = 0.5.
  • Data Augmentation Magnitude: Performance improves across a wide range of increased augmentation magnitudes, indicating robustness of the progressive augmentation strategy.

A.2. Data augmentation in the progressive strategy

The progressive strategy focuses on random scaling because it was empirically the most useful augmentation, while avoiding overly strong transformations that may alter data distribution. Other augmentations, such as mixup, remain possible extensions.

  • Random scaling is used because prior experiments identified it as the most useful augmentation method for semantic segmentation.
  • Very strong brightness or rotation augmentation may influence data distribution, motivating the focus on random scaling.
  • The authors suggest incorporating methods such as mixup into the progressive strategy in future work.

A.3. Visualization of Pseudo Labels

DARS redistributes pseudo labels away from dominant classes and toward tail classes, with iterative labeling further expanding tail-class coverage.

  • ST and CBST pseudo labels are overwhelmed by majority classes such as road.
  • Table 12 compares semi-supervised approaches on ScanNet with tail classes highlighted and per-class best results bolded.
  • The accompanying visualization reports that tail objects such as pole and traffic light are often ignored, leaving pseudo-label distributions biased toward dominant classes.
  • DARS reduces dominant-class percentages and redistributes pseudo labels across larger spatial areas.At round k=1, it also pseudo-labels tail classes such as pole and traffic light.
  • At round k=2, increasing the labeling ratio to 50% further enhances pseudo-label quality and incorporates more tail-class objects.

A.4. Qualitative Results

On Cityscapes, previous methods miss or confuse tail classes, whereas DARS alleviates both failure modes and produces more consistent tail-class segmentation.

  • The comparison evaluates ST, CBST, and DARS on Cityscapes at round k=1.
  • Previous methods omit tail classes such as fence, traffic light, and wall, or confuse tail objects with similar classes.Examples include missing fence, lost traffic light, unrecognized wall, and bus regions mistaken for vegetation, truck, or car.
  • DARS alleviates tail-class omission and confusion, outperforming ST and CBST on tail classes in the qualitative comparison.

B. Additional Experiments on ScanNet

On ScanNet, DARS outperforms ST and CBST with little added computational cost, approaches fully supervised performance using one-quarter labeled data, and gains less from self-training than on Cityscapes.

  • ScanNet evaluation uses 1/8- and 1/4-split labeled settings across 19,466 training and 5,436 validation images.
  • Only one experiment per ScanNet setting is run because reported variance is rather small, with 20 epochs per training round and 481×481 crops.
  • DARS introduces little computational cost compared with the evaluated approaches.
  • 58.35% mIoU is achieved with only 1/4 labeled data, close to the fully supervised result of 61.69%.
  • Self-training yields relatively small gains on ScanNet compared with Cityscapes, attributed mainly to indoor scenes’ greater variance and complex spatial relationships.

C. Unsupervised Domain Adaptation Setting

In GTA5-to-Cityscapes adaptation, smaller pseudo-label distribution mismatch corresponds to better performance, while perfect alignment produces larger gains but relies on target-label information unavailable in the standard setting.

  • The study examines the relationship between pseudo-label distribution mismatch, measured by KL divergence, and performance boost.
  • Without a labeled target-domain set, DARS cannot obtain the true target-label distribution for unbiased pseudo-label generation in the standard comparison setting.
  • The adaptation study uses GTA5 → Cityscapes and compares ST, CBST, DARS (SD), and DARS (TD).
  • Smaller KL divergence between pseudo-label and target-label distributions corresponds to better performance.
  • DARS (TD) achieves 4% higher mIoU than DARS (SD) and 2.27% higher than CBST in a single round.
  • DARS (TD) reaches 55.0% mIoU with iterative training, but the comparison is stated to be unfair because it uses the target distribution from the validation set.
Loading 2107.11279v2…