PaperlayerView exact revision on arXiv

Source-linked AI summary

Semi-supervised Semantic Segmentation with Error Localization Network

Donghyeon Kwon, Suha Kwak

arXiv:2204.02078v3cs.CV

TL;DR

Noisy pseudo labels in semi-supervised semantic segmentation can reinforce their own errors through confirmation bias. The paper uses an error localization network (ELN) to identify and filter likely erroneous pixels, with diverse simulated errors in ELN training. It reports state-of-the-art performance on PASCAL VOC 2012 and Cityscapes across evaluation settings.

  • Problem

    Noisy pseudo labels used for self-training and contrastive learning can reinforce errors through confirmation bias, while uncertainty filtering depends heavily on hand-tuned thresholds.

    Source: §1 · p. 1

  • Method

    ELN identifies likely erroneous pseudo-label pixels for filtering, and auxiliary decoders simulate plausible, diverse prediction errors to improve ELN generalization.

    Source: §1 · pp. 1–2

  • Results

    The method achieves state of the art on PASCAL VOC 2012 and Cityscapes in every evaluation setting.

    Source: §1 · pp. 2–3, §5 · p. 9

  • Takeaways & Limitations

    Error localization can be integrated with self-training and contrastive learning, and the reported experiments validate its use for semi-supervised segmentation on both benchmarks.

    Source: §1 · p. 2, §5 · p. 9

    Training requires relatively more GPU memory because of auxiliary networks, and ELN sometimes misses erroneous predictions when the main network is highly confident.

    Source: §5 · p. 9

Abstract

from arXiv · show

This paper studies semi-supervised learning of semantic segmentation, which assumes that only a small portion of training images are labeled and the others remain unlabeled. The unlabeled images are usually assigned pseudo labels to be used in training, which however often causes the risk of performance degradation due to the confirmation bias towards errors on the pseudo labels. We present a novel method that resolves this chronic issue of pseudo labeling. At the heart of our method lies error localization network (ELN), an auxiliary module that takes an image and its segmentation prediction as input and identifies pixels whose pseudo labels are likely to be wrong. ELN enables semi-supervised learning to be robust against inaccurate pseudo labels by disregarding label noises during training and can be naturally integrated with self-training and contrastive learning. Moreover, we introduce a new learning strategy for ELN that simulates plausible and diverse segmentation errors during training of ELN to enhance its generalization. Our method is evaluated on PASCAL VOC 2012 and Cityscapes, where it outperforms all existing methods in every evaluation setting.

1. Introduction

The introduction frames semi-supervised segmentation around the challenge of learning from noisy pseudo labels without reinforcing their errors. It introduces error localization and diverse-error training for ELN, then reports state-of-the-art results on both benchmarks across evaluation settings.

1. Introduction

  • Pixel-wise annotation is labor-intensive, while noisy pseudo labels can cause confirmation bias and corrupted models; threshold-based uncertainty filtering depends heavily on hand-tuned thresholds.Source: §1 · p. 1
  • Earlier error-correction networks can generalize poorly because a segmenter overfits the small labeled subset and exposes them to too narrow a range of prediction errors.Source: §1 · p. 1
  • ELN identifies pixels with erroneous pseudo labels so training can disregard them rather than correct them, addressing confirmation bias in pseudo-label learning.The paper also characterizes error localization as a class-agnostic, easier subproblem of error correction.Source: §1 · pp. 1–2
  • The framework uses teacher-generated pseudo labels for student self-training and contrastive learning, with ELN filtering likely-noisy pixels in both.Self-training uses the segmentation head, contrastive learning uses the feature-embedding head, and the teacher is updated by an exponential moving average of the student.Source: §1 · p. 2, Figure 1 · p. 2
  • Auxiliary decoders generate plausible, diverse prediction errors for ELN training, improving its generalization even with few labeled examples.They are trained to reach different accuracy levels, simulating the segmentation network at different training stages.Source: §1 · p. 2, Figure 2 · p. 3
  • The method achieves state of the art on PASCAL VOC 2012 and Cityscapes in every evaluation setting.The introduction also reports superior performance to previous work on both datasets while varying the number of labeled training images.Source: §1 · pp. 2–3

2. Related Work

Related work spans methods for semantic segmentation and approaches that reduce its dependence on pixel-labeled data. For semi-supervised segmentation, it reviews adversarial learning, consistency regularization, self-training, contrastive learning, and auxiliary correction networks.

2. Related Work

  • Semantic segmentation methods include encoder-decoder structures, ASPP and PSPNet for spatial context, and attention for global relations, but require substantial data and labor.Source: §2 · p. 3
  • Semi-supervised segmentation uses GAN or adversarial learning, consistency regularization, self-training with pseudo labels, and contrastive learning.Source: §2 · p. 3
  • Auxiliary error-correction networks learn prediction-to-ground-truth differences, but overfitting to labeled data can limit their generalization.Source: §2 · p. 3

3. Proposed Method

The method first trains an error localization network (ELN) with labeled data and simulated segmentation errors, then uses it to exclude likely erroneous pseudo labels during semi-supervised learning. The framework applies ELN to both self-training and contrastive learning.

3. Proposed Method

  • ELN identifies likely erroneous pseudo-label pixels, which the framework disregards in both self-training and contrastive learning.The method has two stages: learning ELN from labeled data, followed by semi-supervised learning with ELN.Source: §3 · pp. 3–4

3.1. Learning ELN Using Labeled Data

  • Auxiliary decoders are constrained to produce plausible erroneous predictions, giving ELN diverse error patterns to learn from; incorrect-pixel losses are reweighted because correct-pixel labels dominate.ELN is trained to identify errors by comparing segmentation predictions with ground truth.Source: §3.1 · p. 4

3.2. Semi-supervised Learning with ELN

  • A mean-teacher setup supplies pseudo labels, while ELN restricts self-training to valid pixels and contrastive learning uses valid student–teacher embeddings.The contrastive objective attracts features with the same pseudo label and separates features from different categories, considering relations across the input batch.Source: §3.2 · pp. 4–5

4. Experiments

Experiments evaluate the method on PASCAL VOC 2012 and Cityscapes across labeled-data ratios, reporting stronger results on both benchmarks. Ablations examine the auxiliary decoders and training losses.

4.1. Network Architecture

  • The segmentation network uses DeepLab v3+ with a ResNet backbone, while ELN uses ResNet-34 and takes the image concatenated with tensors as input.The main network uses ResNet-50 or ResNet-101; both networks have encoders and decoders, with auxiliary decoders attached to the main segmentation network.Source: §4.1 · p. 5

4.3. Results

  • Across PASCAL VOC 2012 and Cityscapes, the method outperforms prior methods across tested labeled-data ratios and both tested PASCAL backbones.The PASCAL VOC 2012 results cover ResNet-50 and ResNet-101; Cityscapes results cover three labeled-data ratios.Source: Table 1 · p. 5, Table 2 · p. 5, §4.3 · p. 6

4.2. Implementation Details

  • Evaluation uses PASCAL VOC 2012 and Cityscapes, reports mIoU, and varies labeled-data ratios from 1/20 to 1/4 and 1/8 to 1/2, respectively.PASCAL images are resized to 512×512 for training; Cityscapes images are randomly cropped to 512 × 1024.Source: §4.2 · pp. 5–6

4.3. Results

  • ELN achieved the highest mIoU and error-localization F1 among ELN, s-ECN, and confidence-threshold approaches.The mIoU comparison uses the PASCAL VOC 2012 validation set; the F1 comparison evaluates error localization on unlabeled data at a 1/20 ratio.Source: Table 3 · p. 6, Table 4 · p. 6, §4.3 · p. 6

4.4. Ablation Studies

  • More auxiliary decoders and a higher loss-constraint value improved performance; two decoders already yielded sufficient gains.The ablation varies decoder count and the constraint parameter α.Source: §4.4 · pp. 7–8, Table 5 · p. 8
  • Each loss term contributed to performance, and jointly using the pseudo-label and contrastive losses produced the strongest result.The comparison includes a confidence-threshold method that uses neither ELN-associated loss.Source: §4.4 · pp. 8–9, Table 6 · p. 9

5. Conclusion

The framework uses ELN and its training scheme to mitigate confirmation bias from invalid pseudo labels. The authors report state-of-the-art results on PASCAL VOC 2012 and Cityscapes, alongside memory and confidence-related limitations.

  • The method achieved state of the art on both PASCAL VOC 2012 and Cityscapes with high generalization capability.Source: §5 · p. 9
  • ELN and its training scheme mitigate performance degradation from confirmation bias, and experiments validate its removal of pseudo-label errors on unseen data.The experiments also show that simulating erroneous predictions with auxiliary decoders is helpful.Source: §5 · p. 9
  • Training requires relatively more GPU memory because of auxiliary networks, with memory use increasing as auxiliary decoders are added.ELN sometimes fails to identify erroneous predictions when the main segmentation network is highly confident, or low-entropy.Source: §5 · p. 9

Related papers

Loading 2204.02078v3…