PaperlayerView exact revision on arXiv

Source-linked AI summary

PseudoSeg: Designing Pseudo Labels for Semantic Segmentation

Yuliang Zou, Zizhao Zhang, Han Zhang, Chun-Liang Li, Xiao Bian, Jia-Bin Huang, Tomas Pfister

arXiv:2010.09713v2cs.CV

TL;DR

Pixel-level annotation is costly, and segmentation performance drops sharply with limited labeled data while existing SSL strategies face structured-output challenges. PseudoSeg redesigns pseudo-labeling within a one-stage consistency framework using unlabeled or weakly labeled data. The method improves segmentation in both low- and high-data regimes, with experiments emphasizing fused, well-calibrated soft pseudo labels and strong augmentation.

  • Problem

    Pixel-level segmentation labels are expensive, and limited labeled data causes a significant performance drop while making existing SSL pseudo-labeling and augmentation strategies difficult to apply.

  • Method

    PseudoSeg uses a one-stage, network-agnostic consistency framework that fuses diverse predictions into structured pseudo labels and trains strongly augmented inputs to match them.

  • Results

    PseudoSeg consistently outperforms supervised training on VOC12 and improves performance on COCO, including low-data and full-data settings.

  • Takeaways & Limitations

    Well-calibrated soft pseudo labels from fused diverse sources and strong augmentation are central to effective consistency training for segmentation.

  • Takeaways & Limitations

    The method is illustrated first for unlabeled data and requires adaptation to use image-level labeled data.

Abstract

from arXiv · show

Recent advances in semi-supervised learning (SSL) demonstrate that a combination of consistency regularization and pseudo-labeling can effectively improve image classification accuracy in the low-data regime. Compared to classification, semantic segmentation tasks require much more intensive labeling costs. Thus, these tasks greatly benefit from data-efficient training methods. However, structured outputs in segmentation render particular difficulties (e.g., designing pseudo-labeling and augmentation) to apply existing SSL strategies. To address this problem, we present a simple and novel re-design of pseudo-labeling to generate well-calibrated structured pseudo labels for training with unlabeled or weakly-labeled data. Our proposed pseudo-labeling strategy is network structure agnostic to apply in a one-stage consistency training framework. We demonstrate the effectiveness of the proposed pseudo-labeling strategy in both low-data and high-data regimes. Extensive experiments have validated that pseudo labels generated from wisely fusing diverse sources and strong data augmentation are crucial to consistency training for segmentation. The source code is available at https://github.com/googleinterns/wss.

1 INTRODUCTION

Semantic segmentation becomes substantially less effective when pixel-level labels are scarce, motivating data-efficient training with unlabeled or weakly annotated data. PseudoSeg addresses segmentation-specific SSL challenges through a one-stage framework centered on redesigned pseudo-labeling and consistency training.

  • Pixel-accurate segmentation annotation is dramatically more expensive than annotation for other vision tasks.
  • Limited pixel-labeled data causes segmentation performance to drop significantly and hinders model applicability.
  • Consistency training uses predictions from weakly augmented images as pseudo labels for strongly augmented counterparts in SSL classification.
  • Segmentation requires carefully designed pseudo labels and augmentation because existing SSL strategies are non-trivial to transfer directly.
  • PseudoSeg is a one-stage, architecture-agnostic framework that learns from limited pixel labels plus unlabeled or image-level labeled data.

2 RELATED WORK

Prior work in semi-supervised segmentation uses unlabeled data through generative, consistency-based, dual-branch, and iterative self-training approaches. Weakly supervised segmentation commonly exploits image-level labels and class activation maps, while Figure 1 depicts a consistency-training branch combining decoder and SGC predictions.

  • Semi-supervised semantic segmentation: Semi-supervised segmentation methods have used GANs, augmented inputs, perturbed features, different networks, coarse labels, and iterative self-training to exploit additional data.
  • Semi-supervised semantic segmentation: Iterative self-training methods generally assume enough labeled data to train a teacher model that generates pseudo labels.
  • Weakly-supervised semantic segmentation: Weakly supervised segmentation often uses image-level labels with class activation maps, whose initial maps focus on the most discriminative object regions.
  • PseudoSeg training branch: Figure 1 shows weakly augmented inputs producing decoder and SGC predictions, which are fused into pseudo labels for strong-augmentation consistency training.

3 THE PROPOSED METHOD

PseudoSeg combines supervised pixel-wise learning with consistency training on additional data. Its structured pseudo labels fuse decoder predictions with refined localization maps, while strong augmentation trains predictions to match them.

  • Training objective: PseudoSeg optimizes supervised pixel-wise cross-entropy on labeled data alongside a consistency loss on unlabeled data.The supervised branch uses weak augmentation, while the unsupervised branch applies pseudo labels to strongly augmented inputs.
  • Training objective: The framework estimates structured pseudo labels for additional data and uses them to supervise predictions from strongly augmented counterparts.The same framework can use unlabeled or image-level labeled data.
  • Structured pseudo labels: PseudoSeg introduces Grad-CAM-based localization because localization requires coarser outputs than precise segmentation.The method uses Grad-CAM as a distinct decision mechanism alongside decoder predictions.
  • Structured pseudo labels: Self-attention Grad-CAM propagates high Grad-CAM responses to adjacent regions using similarities between hypercolumn features.A skip connection preserves the original regional response while the learned function performs propagation.
  • Structured pseudo labels: The calibrated fusion strategy combines decoder and SGC predictions to exploit the decoder’s boundaries and SGC’s locally consistent masks.SGC maps are low-resolution and therefore insufficient alone for crisp boundaries; normalization and sharpening address calibration during fusion.
  • Additional data and augmentation: Image-level labels can provide Grad-CAM supervision and additional classification supervision within the one-stage framework.Strong augmentation uses color jittering and random CutOut, with geometric transformations removed.

4 EXPERIMENTAL RESULTS

Experiments on VOC12 and COCO evaluate PseudoSeg with unlabeled or image-level labeled data, across low- and high-data regimes. Ablations examine pseudo-label construction, augmentation, feature choices, and backbone architectures.

  • Experiments using pixel-level labeled data and unlabeled data: PseudoSeg consistently outperforms supervised baselines on VOC12 and COCO when using unlabeled data, including the 6.25% pixel-label regime and the full 1.4k-image training set.The VOC12 improvement is especially large with only 6.25% pixel-level labels, while gains also appear with the entire training set.
  • Experiments using pixel-level labeled data and image-level labeled data: With image-level labeled data, PseudoSeg consistently improves strong baselines on both datasets and is more robust than the compared method across low-data splits.Across all tested splits on both datasets, image-level labeled data produces higher mIoU than using pixel-level labeled data with unlabeled data.
  • Ablation study: The proposed fusion of decoder and self-attention Grad-CAM predictions outperforms using either source alone, with or without additional image-level labels.The ablation compares alternative pseudo-labeling strategies on 1/4, 1/8, and 1/16 pixel-level-data splits.
  • Ablation study: Using hypercolumn features substantially improves performance, while all-prediction soft pseudo labels outperform selecting only confident predictions.These results support the design choices of hypercolumn-based refinement and soft rather than thresholded pseudo labels.
  • Ablation study: Temperature sharpening consistently improves results, and sufficiently strong color-jitter augmentation is important because weak jitter causes a significant performance drop.The augmentation ablation uses pixel-level labeled data with unlabeled data, whereas the temperature-sharpening ablation uses image-level labeled data.
  • Ablation study: PseudoSeg improves the supervised baseline by a substantial margin across ResNet-50, ResNet-101, and Xception-65 backbones.The comparison is conducted using pixel-level labeled data and unlabeled data.

5 DISCUSSION AND CONCLUSION

PseudoSeg’s low-data performance depends on redesigned pseudo-labeling that remedies weak segmentation predictions, while augmentation consistency progressively improves the segmentation head. The SGC pipeline constructs attention-based maps from backbone features and projected key-query embeddings.

  • Discussion and Conclusion: Well-calibrated soft pseudo labels allow unlabeled or image-level labeled data to improve segmentation in low-data regimes.The paper presents this as a demonstrated result and identifies stronger, better-calibrated fusion as future work.
  • Discussion and Conclusion: Color jittering works as strong augmentation, whereas extensively explored geometric augmentations using STN alignment were not helpful.The authors argue that segmentation augmentation needs reconsideration beyond approaches successful in classification.
  • Discussion and Conclusion: SGC constructs hypercolumn features from the backbone’s last two convolutional stages, then projects them into key and query embeddings for attention computation.The SGC map-generation pipeline is illustrated in Figure 6.

B IMPLEMENTATION DETAILS

The implementation uses the official DeepLab codebase with DeepLabv3+ and an ImageNet-pretrained Xception-65 backbone. Training follows specified default hyperparameters on 16 GPUs.

  • Implementation Details: The method is implemented on the official DeepLab codebase using DeepLabv3+ with Xception-65 as the ImageNet-pretrained feature backbone.These are the default model and backbone choices unless otherwise specified.
  • Implementation Details: Training uses an initial learning rate of 0.007 with polynomial decay, 513 × 513 crops, and encoder output stride 16.The listed settings are adopted as default hyperparameters.
  • Implementation Details: The model is trained using 16 GPUs.This hardware setting is part of the implementation configuration.

C LOW-DATA SAMPLING IN PASCAL VOC 2012

Low-data VOC sampling addresses rare-class imbalance by repeating random sampling while ensuring class coverage. The section reports results for Split 1 and releases all three splits for reproducibility.

  • Low-Data Sampling in PASCAL VOC 2012: Rare-class imbalance makes uniform sampling difficult in low-data semantic segmentation.The sampling issue differs from random sampling in image classification because extremely small subsets may omit classes.
  • Low-Data Sampling in PASCAL VOC 2012: The authors repeat the 1/16 random sampling process three times while ensuring each class has a certain amount.This procedure is intended to avoid missing classes in extremely low-data regimes.
  • Low-Data Sampling in PASCAL VOC 2012: The main manuscript reports Split 1, while all three splits are released to encourage reproducibility.The accompanying figure presents training supervision and corresponding data for each network component, and Table 8 gives full Split 1 results.

D HIGH-DATA EXPERIMENTAL SETTINGS

The high-data experiments benchmark weakly supervised semantic segmentation against methods using VOC12 image-level labels. They use an extended training schedule and note that this schedule does not improve the supervised baseline.

  • High-Data Experimental Settings: High-data experiments use a longer 90,000-iteration training schedule because substantially more unlabeled or image-level labeled data are available.The fusion strategy in this setting uses T = 0.7 and γ = 0.3.
  • High-Data Experimental Settings: The weakly supervised benchmark uses VOC12 training and augmented sets, with all methods using image-level labels.Table 9 compares recent weakly supervised semantic segmentation methods on the PASCAL VOC 2012 validation set.
  • High-Data Experimental Settings: The longer training schedule does not improve the supervised baseline.This qualification bounds interpretation of schedule changes in the high-data setting.

F PERFORMANCE ANALYSIS FOR TEMPERATURE SHARPENING

The analysis examines how the temperature parameter T affects distribution sharpening on the VOC 1/16 split. Using T < 1 generally improves performance.

  • T < 1 generally leads to improved performance with distribution sharpening.The experiments use the VOC 1/16 split with pixel-level labeled and image-level labeled data.
  • The analysis evaluates performance across different temperature values T.

G EXPERIMENTS ON CITYSCAPES

Cityscapes experiments evaluate PseudoSeg with pixel-level labeled and unlabeled data, including comparisons with a state-of-the-art method and per-class analysis. The method improves the supervised baseline across most classes and substantially across backbone architectures in the reported settings.

  • Experiments on Cityscapes: Cityscapes contains 2,975 training frames and 500 validation frames from 50 real-world driving sequences, evaluated on 19 semantic classes.
  • Comparison with state of the art: The method is compared with French et al. (2020) using 1/4, 1/8, and 1/30 pixel-level labeled splits, with ResNet-101 and DeepLabv3+.
  • Per-class performance analysis: PseudoSeg improves upon the supervised baseline on most classes in the 1/30 split, except Wall and Pole, despite severely imbalanced labeled-pixel distributions.
  • Discussion: The discussion notes that aggressive geometric augmentation can generate diverse image-level labels from similar full-image layouts.
  • Discussion: Diverse Internet images can be collected and weakly labeled instead of weakly labeling images from Cityscapes.

H QUALITATIVE RESULTS

Qualitative results compare supervised and PseudoSeg predictions in low-data settings on PASCAL VOC 2012 and COCO. PseudoSeg produces more satisfying predictions, particularly where the supervised baseline struggles with categories and small objects.

  • Qualitative comparison: The supervised baseline struggles with some categories and small objects in the low-data regime, while PseudoSeg uses unlabeled or weakly labeled data to generate more satisfying predictions.
  • PASCAL VOC 2012: On PASCAL VOC 2012, models are trained with 1/16 pixel-level labeled data.
  • COCO: On COCO, models are trained with 1/512 pixel-level labeled data, and white ground-truth pixels are not annotated for evaluation.
Loading 2010.09713v2…