PaperlayerView exact revision on arXiv

Source-linked AI summary

Semi-Supervised Semantic Segmentation with Cross-Consistency Training

Yassine Ouali, Céline Hudelot, Myriam Tami

arXiv:2003.09005v3cs.CV

TL;DR

Semantic segmentation requires costly pixel-level labels, while existing semi-supervised approaches have limitations in exploiting unlabeled data or training difficulty. The paper proposes cross-consistency training, which enforces agreement between a main decoder and auxiliary decoders over perturbed encoder outputs, and reports competitive results across datasets and settings.

  • Problem

    Pixel-level annotation is costly, while existing semantic-segmentation approaches either require weak labels alongside pixel labels or can be harder to train.

  • Method

    CCT trains a shared encoder and main decoder with labeled data, then enforces consistency between main and auxiliary decoder predictions on perturbed encoder outputs from unlabeled data.

  • Results

    CCT obtains competitive results across PASCAL VOC, CityScapes, CamVid, and SUN under semi-supervised and semi-supervised domain-adaptation settings.

  • Takeaways & Limitations

    CCT is presented as a simple, efficient, and flexible consistency-based method that can incorporate weak labels and pixel-level labels across domains.

  • Takeaways & Limitations

    Training samples equal numbers of labeled and unlabeled examples, causing repeated iterations over the labeled set and risking overfitting.

Abstract

from arXiv · show

In this paper, we present a novel cross-consistency based semi-supervised approach for semantic segmentation. Consistency training has proven to be a powerful semi-supervised learning framework for leveraging unlabeled data under the cluster assumption, in which the decision boundary should lie in low-density regions. In this work, we first observe that for semantic segmentation, the low-density regions are more apparent within the hidden representations than within the inputs. We thus propose cross-consistency training, where an invariance of the predictions is enforced over different perturbations applied to the outputs of the encoder. Concretely, a shared encoder and a main decoder are trained in a supervised manner using the available labeled examples. To leverage the unlabeled examples, we enforce a consistency between the main decoder predictions and those of the auxiliary decoders, taking as inputs different perturbed versions of the encoder's output, and consequently, improving the encoder's representations. The proposed method is simple and can easily be extended to use additional training signal, such as image-level labels or pixel-level labels across different domains. We perform an ablation study to tease apart the effectiveness of each component, and conduct extensive experiments to demonstrate that our method achieves state-of-the-art results in several datasets.

1. Introduction

The paper addresses the high cost of pixel-level annotation in semantic segmentation by proposing cross-consistency training, which exploits unlabeled data through perturbations of encoder outputs. The method is extended to weak labels and cross-domain pixel labels, with experiments reporting competitive results across datasets and settings.

  • Pixel-level labels for semantic segmentation cost 15 times more than region-level labels and 60 times more than image-level labels.
  • Cross-consistency training enforces prediction invariance across perturbations of the encoder’s output rather than the input.A shared encoder and main decoder use labeled examples, while auxiliary decoders process perturbed encoder outputs for unlabeled examples.
  • The approach studies various perturbation types and their combined effectiveness through an exhaustive ablation analysis.
  • CCT extends to weakly labeled data and pixel-level labels across different domains.
  • Experiments compare CCT with state-of-the-art methods and evaluate it across multiple datasets and training settings.

2. Related Work

Prior semi-supervised segmentation methods use weak labels, adversarial training, or other consistency strategies, but they either require additional annotations or can be difficult to train. CCT instead applies consistency regularization to hidden representations and extends it across domains, including disjoint label spaces.

  • Unlike input-perturbation consistency methods such as CowMix, CCT perturbs encoder outputs and uses auxiliary decoders.
  • Weakly supervised segmentation combines limited pixel-level labels with region-level or image-level annotations to generate training targets.
  • Adversarial segmentation methods exploit unlabeled examples through pixel-level discriminator predictions and adversarial loss.
  • CCT enforces consistency across multiple perturbations of hidden representations, with small additional computation and memory requirements.
  • CCT applies consistency across domains and reports better generalization even with non-overlapping label spaces.

3. Method

The method uses a shared encoder, supervised main decoder, and auxiliary decoders to enforce prediction consistency across perturbed encoder outputs for unlabeled images. It motivates hidden-representation perturbations because class boundaries align with low-density regions there, unlike at the input level.

  • 3.1. The cluster assumption in semantic segmentation: Hidden representations preserve the cluster assumption: class boundaries have high average distance and correspond to low-density regions, unlike input-level representations.The analysis compares local distances between neighboring input patches with distances between neighboring 2048-dimensional activations.
  • 3.2.1 Problem Definition: The segmentation network combines a shared encoder h and main decoder g, while K auxiliary decoders enforce consistency on unlabeled examples.The main network is f = g◦h; auxiliary decoders receive perturbed encoder outputs while the main decoder receives the uncorrupted representation.
  • 3.2.2 Cross-Consistency Training: For labeled images, the main decoder is trained with pixel-level labels and cross-entropy loss; unlabeled images provide encoder representations for consistency training.The supervised loss uses labeled examples and the main encoder output, while unlabeled examples are processed through perturbed intermediate representations.
  • 3.2.3 Perturbation functions: Auxiliary decoders receive K perturbed representations generated from R stochastic perturbation functions, including feature-based and prediction-based perturbations.Feature perturbations inject noise or drop activations; prediction-based perturbations include guided masking, guided cutout, and adversarial perturbations.
  • 3.2.2 Cross-Consistency Training: The unsupervised loss minimizes mean squared error between main-decoder and auxiliary-decoder probability predictions for perturbed encoder outputs.The total objective combines supervised and unsupervised losses, with the unsupervised weight ramping from zero to a fixed value.
  • 3.2.3 Perturbation functions: Feature perturbations inject proportional noise or mask 10% to 40% of the most active feature-map regions, while guided methods mask objects, context, or object crops.F-Noise uses N ∼ U(−0.3, 0.3); F-Drop samples γ ∼ U(0.6, 0.9) and masks the most active regions.

4. Experiments

Experiments evaluate CCT through ablations, supervised comparisons, and semi-supervised domain adaptation. Across CamVid, PASCAL VOC, and cross-domain settings, the reported results show gains over labeled-only baselines and flexibility with additional labels.

  • Ablation studies: Ablations vary perturbation types, auxiliary-decoder counts K, combined perturbations, and annealed-bootstrapped CE against a labeled-only baseline.CCT full uses all seven perturbations, with K × 7 auxiliary decoders.
  • CamVid ablation: Up to 21 points separate perturbations from the baseline in CamVid’s 20-label setting.Each perturbation outperforms the baseline, while increasing K generally yields modest improvement.
  • PASCAL VOC ablation: 10 to 15 points of improvement over the baseline are reported for different perturbations on PASCAL VOC with 1000 labeled examples.Combining perturbations adds a small increase, and annealed-bootstrapped CE adds up to 7 points over CCT full.
  • Comparison to previous work: CCT outperforms previous semi-supervised methods on PASCAL VOC, including methods using image-level labels.Additional image-level labels increase performance by 3.8 points.
  • Semi-supervised domain adaptation: The domain-adaptation experiments also evaluate Cityscapes with SUN RGB-D under disjoint label spaces and distribution mismatch.This setting tests whether CCT can extract visual relationships and transfer knowledge between dissimilar domains.

5. Conclusion

The paper presents cross-consistency training (CCT) as a simple, efficient, and flexible method for semi-supervised semantic segmentation, reporting state-of-the-art results. Future work could explore perturbations at other network levels and applications to other visual tasks and learning settings.

  • CCT is introduced as a simple, efficient, and flexible consistency-based method for semi-supervised semantic segmentation.
  • The method achieves state-of-the-art results in semantic segmentation.
  • Future work may investigate perturbations applied at different levels within the segmentation network.
  • Future work may also examine CCT in other visual tasks and learning settings, including unsupervised domain adaptation.

A. Comparison with Traditional Consistency Training Methods

This comparison tests whether consistency over hidden representations better fits semantic segmentation than consistency over inputs. The reported results favor CCT over traditional input-based consistency methods.

  • Results: VAT with adversarial noise applied to inputs performs below the baseline, while Mean Teachers achieve performance similar to the baseline.The comparison is conducted on PASCAL VOC with varying numbers of labeled examples.
  • Interpretation: The reported comparison supports enforcing consistency over encoder outputs rather than inputs for semantic segmentation.The paper connects this design to better alignment with the cluster assumption and improved results.

B.1. Distance Measures

The experiments compare consistency-distance choices and several extensions of CCT across semi-supervised and domain-adaptation settings. MSE and JS perform similarly, while KL and additional adversarial or pairwise objectives introduce specific drawbacks.

  • Distance measures: MSE and JS achieve similar performance, whereas KL provides only 2.6 and 3.3 points gain over baseline for n = 500 and n = 1000.The comparison is conducted on PASCAL VOC.
  • Distance measures: KL can penalize sharp but incorrect auxiliary predictions, pushing them toward uniform outputs and reducing unlabeled training signal.JS avoids this through symmetrization and smoothing; MSE is less sensitive to completely incorrect predictions and has lower computational cost.
  • Additional losses: Confidence masking does not improve CCT, while adding pairwise auxiliary consistency lowers performance by 3 and 3.2 points in the two settings.The results indicate that uncertain main predictions do not hinder performance, whereas the pairwise term may constrain auxiliary decoders undesirably.
  • CCT procedure: CCT uses a shared encoder, a main decoder, and K auxiliary decoders that receive perturbed encoder outputs and are trained through supervised and unsupervised losses.The algorithm forwards labeled data through the encoder and main decoder, then applies perturbations before auxiliary decoding on unlabeled data.
  • Domain adaptation: Adding a discriminator branch to align encoder representations across domains diminishes segmentation performance, suggesting a learning conflict with CCT.The discriminator predicts domain identities from encoder representations within a min-max framework.
  • Inference: Multi-scale inference evaluates ten transformed versions per image and aggregates rescaled predictions by pixel-wise average pooling before taking class-wise argmax.The ten versions combine five scales with horizontal flips.

G. Virtual Adversarial Training (VAT)

The VAT variant applies adversarial perturbations to encoder representations rather than inputs, choosing noise that most changes auxiliary predictions. This improves local smoothness but requires extra passes, with decoder size reducing the added cost.

  • Perturbation design: I-VAT applies VAT perturbations to the encoder output z instead of the unlabeled input.The perturbation is injected before auxiliary decoding.
  • Perturbation design: The adversarial perturbation is estimated by adding Gaussian noise, differentiating the prediction discrepancy, then normalizing and scaling the gradient by ϵ.The discrepancy uses KL-divergence between predictions with and without injected noise.
  • Computational cost: I-VAT requires multiple forward and backward passes per iteration, but its computational burden is reduced because the auxiliary decoders are small.The final perturbed representation is ˜z = radv + z.

H. Dataset sizes

The paper reports the sizes of the dataset splits used in its semantic-segmentation experiments. These split sizes are provided in Table 10.

  • Dataset splits: The dataset-size information covers the experimental splits used throughout the paper.The paper explicitly references the sizes of each split used in its experiments.
  • Dataset splits: Table 10 lists the semantic-segmentation datasets used in the experiments and the size of each split.The passage directs readers to Table 10 for the split sizes.
  • Dataset splits: Table 10 is the paper’s designated reference for comparing dataset split sizes.No individual numerical split sizes are stated in the supplied passages.

I. Further Experimental Details

The paper fixes an encoder-decoder implementation, uses practical inference and manually selected hyperparameters, and ramps unsupervised objectives and confidence thresholds during training. It also reports computational differences among auxiliary decoders.

  • Architecture: Experiments use a ResNet 50 with a PSP encoder and 1 × 1 plus PixelShuffle decoder layers that restore feature maps to the original size.The architecture details are summarized in Table 11.
  • Inference settings: PASCAL VOC ablations use 240 × 240 crops, while state-of-the-art comparisons use 321 × 321 crops and inference on original-sized images.The larger-side resize settings differ between ablations and benchmark comparisons.
  • Hyperparameters: The study avoids intensive hyperparameter search and selects settings by hand for stable training, while acknowledging that comprehensive search may improve performance.The chosen hyperparameters are summarized in Table 12.
  • Ramp-up functions: The unsupervised-loss weight ramps from 0 to λu using an exponential schedule, with t denoting iteration and T the ramp-up length.The example ramp-up length is the first 10% of training time.
  • Ramp-up functions: The ab-CE threshold increases from 1/C to α during ramp-up, using a log schedule that rises quickly at the beginning.The example final threshold is α = 0.9 within the first 40% of training time.
  • Computational statistics: I-VAT has substantially higher computational cost because its perturbations require multiple forward and backward passes, so experiments reduce its decoder count to K = 2.Table 13 compares computation and memory statistics across auxiliary-decoder types.

N. Qualitative Results

The qualitative results examine generated pseudo pixel-level labels and CCT predictions on PASCAL VOC images. High-attention regions generally receive labels corresponding to true positives, while CCT outputs are shown across different n settings.

  • Pseudo pixel-level labels: High-attention regions with scores above 0.3 generally receive pseudo labels corresponding to true positives.These qualitative labels are generated using available image-level labels.
  • Pseudo pixel-level labels: White regions in the generated pseudo pixel-level labels indicate ignored pixels.
  • CCT results: CCT qualitative results on PASCAL VOC validation images compare predictions across different values of n.The results are presented in Fig. 10.
  • CCT results: The displayed comparisons include input images, ground truth, pseudo labels, and CCT outputs for n=1k, n=1.5k, and n=1.5k + 9k weak.
Loading 2003.09005v3…