PaperlayerView exact revision on arXiv

Source-linked AI summary

Pixel Contrastive-Consistent Semi-Supervised Semantic Segmentation

Yuanyi Zhong, Bodi Yuan, Hong Wu, Zhiqiang Yuan, Jian Peng, Yu-Xiong Wang

arXiv:2108.09025v1cs.CV

TL;DR

Semi-supervised segmentation must learn effectively from limited pixel annotations while using abundant unlabeled data, but pixel contrastive learning adds computational and false-negative challenges. PC2Seg jointly enforces label-space consistency and feature-space contrastiveness with pixel-wise ℓ2 and pixel contrastive losses, using negative sampling techniques. It achieves state-of-the-art performance across VOC, Cityscapes, and COCO semi-supervised benchmarks.

  • Problem

    Limited pixel annotations cause segmentation performance to drop, while pixel contrastive learning faces high computation and harmful false negatives.

  • Method

    PC2Seg jointly trains augmentation-consistent output masks and contrastive intermediate pixel features, with negative sampling for pixel contrastive learning.

  • Results

    PC2Seg achieves state-of-the-art performance across multiple semi-supervised segmentation benchmarks, including VOC, Cityscapes, and COCO.

  • Takeaways & Limitations

    Joint pixel consistency and contrastiveness are presented as a promising direction for improving semi-supervised semantic segmentation.

Abstract

from arXiv · show

We present a novel semi-supervised semantic segmentation method which jointly achieves two desiderata of segmentation model regularities: the label-space consistency property between image augmentations and the feature-space contrastive property among different pixels. We leverage the pixel-level L2 loss and the pixel contrastive loss for the two purposes respectively. To address the computational efficiency issue and the false negative noise issue involved in the pixel contrastive loss, we further introduce and investigate several negative sampling techniques. Extensive experiments demonstrate the state-of-the-art performance of our method (PC2Seg) with the DeepLab-v3+ architecture, in several challenging semi-supervised settings derived from the VOC, Cityscapes, and COCO datasets.

1 University of Illinois at Urbana-Champaign 2 X, The Moonshot Factory

The listed affiliations include the University of Illinois at Urbana-Champaign and X, The Moonshot Factory.

  • The authors list the University of Illinois at Urbana-Champaign as an affiliation.
  • The affiliations also include X, The Moonshot Factory.
  • The passage provides affiliation information rather than research findings.

1. Introduction

The paper addresses limited-label semantic segmentation by jointly enforcing label-space consistency and feature-space contrastiveness. PC2Seg combines pixel-wise consistency and contrastive learning while targeting the computational and false-negative challenges of pixel contrastive loss.

  • Semi-supervised segmentation uses unlabeled data to reduce dense annotation requirements in low-data regimes.Limited labeled data can cause severe performance drops because deep models over-fit.
  • PC2Seg jointly enforces augmentation consistency in label space and discriminative pixel representations in feature space.The method uses a pixel-wise ℓ2 consistency loss and a modified pixel-level InfoNCE contrastive loss.
  • Pixel contrastive learning introduces high computational cost and harmful false negatives because segmentation contrasts many pixels.False negatives occur when same-class pixels are mistakenly selected as contrasting examples.
  • The method introduces negative-sampling techniques, including four variants, to improve pixel contrastive learning efficiency and effectiveness.The framework is presented as a single-stage semi-supervised approach using standard losses and data augmentations.
  • The paper reports state-of-the-art performance across multiple semi-supervised semantic-segmentation benchmarks.The contribution is evaluated across widely used benchmarks rather than a single dataset.

2. Related Work

Related work spans contrastive representation learning, semi-supervised consistency methods, and semantic segmentation approaches. The paper positions PC2Seg as pixel-level contrastive-consistent learning for single-stage semi-supervised segmentation.

  • Contrastive Learning: Contrastive learning distinguishes similar image pairs from dissimilar negative pairs using strategies such as memory buffers or large batches.Prior work also studies false-negative bias and consistency training without negatives.
  • Contrastive Learning: Pixel-level contrastive pretraining has shown better transfer to segmentation than image-level self-supervised learning in prior studies.PC2Seg instead applies pixel-level contrastive learning in the semi-supervised setting.
  • Contrastive Learning: Prior supervised segmentation studies use pixel contrasting, whereas PC2Seg focuses on single-stage semi-supervised training.The paper jointly optimizes contrastive loss on intermediate features and consistency loss on output masks.
  • Semi-Supervised Learning: Semi-supervised learning combines labeled and large unlabeled datasets, often using consistency regularization or iterative pseudo-labeling.Related approaches also add self-supervised auxiliary tasks on unlabeled images.
  • Semantic Segmentation: Semantic segmentation predicts pixel-level category labels, commonly using fully convolutional networks trained on large datasets.PC2Seg builds on DeepLab-v3+ following prior semi-supervised segmentation work.
  • Semi-Supervised Semantic Segmentation: Other semi-supervised segmentation methods use adversarial training, augmented-view consistency, weak labels, or unsupervised clustering.These approaches address annotation reduction under different problem settings.

3. Method

PC2Seg trains on labeled and unlabeled streams, combining supervised segmentation with consistency between weak and strong views and pixel-level contrastive learning. Its contrastive design reduces computation and false negatives through projection and alternative negative-sampling strategies.

  • Overview: PC2Seg uses shared network components for labeled and unlabeled streams, applying supervised cross-entropy to labeled images.The streams share an FCN and decoder for feature extraction and mask prediction.
  • Consistency Learning: Weak and strong augmentations of each unlabeled image provide pseudo labels and a pixel-wise consistency objective for the predicted masks.Weak-branch predictions are sharpened and stop-gradient pseudo labels, while gradients update the strong branch.
  • Pixel Contrastive Loss: Pixel contrastive learning matches corresponding pixels across augmented views and contrasts them using an InfoNCE loss with cosine similarity.The loss operates on intermediate feature vectors, with temperature τ set to 0.07 in the experiments.
  • Pixel Contrastive Loss: A linear projection reduces the feature dimension from 2,048 to 128 to lower the memory and computation required for pixel contrastive learning.The high-dimensional example is a 33 × 33 × 2,048 DeepLab backbone feature map.
  • Negative Sampling Strategies: Negative sampling addresses pixel-level contrastive learning challenges by limiting candidate negatives and reducing harmful false negatives.The paper investigates uniform, different-image, and pseudo-label-debiased sampling among four strategies.
  • Negative Sampling Strategies: Different Image excludes candidates from the anchor’s image, while Pseudo-Label Debiased sampling weights candidates by predicted class disagreement.The latter uses model predictions to favor pixels believed to belong to different categories.

4. Experiment

PC2Seg performs strongly across VOC, Cityscapes, and COCO semi-supervised settings, while analyses show that its sampling, loss, and contrastive-layer choices affect accuracy, noise, and cost.

  • VOC 2012: PC2Seg outperforms prior methods in almost all VOC 2012 labeled splits with ResNet-50 and ResNet-101 backbones.The 1/16 split uses only 92 labeled images and reports results over three random splits.
  • VOC 2012: 74.15% mIoU with ResNet-101 is achieved on the VOC 1.4k/9k split, exceeding prior methods.
  • Cityscapes: 0.84% mIoU separates the full-set result of 75.99% from the 1/4 labeled Cityscapes result of 75.15% with ResNet-101.PC2Seg outperforms the supervised baseline across all semi-supervised Cityscapes settings.
  • COCO: PC2Seg performs better than PseudoSeg in all COCO labeled splits, and its 1/8 result nearly matches supervised full-data performance.The method also gains over the supervised baseline in the full-data setting.
  • Analyses: Different Image + Pseudo Label sampling achieves mIoU almost as good as the Oracle strategy while reducing false-negative noise relative to simpler sampling.False-negative rates decrease with more complex sampling distributions, whereas Uniform performs worst.
  • Analyses: 64.63% mIoU from joint contrastive-consistent learning exceeds 63.75% for purely consistent learning and 53.53% for purely contrastive learning.The adopted loss coefficients are λ1 = 0.3 and λ2 = 1.
  • Analyses: PC2Seg training time is comparable to PseudoSeg, and removing the contrastive component reduces training time by less than 5min.The loss ablation also identifies label ℓ2 plus feature pixel contrastive loss as the best combination, while a mid-level feature layer performs best.

5. Conclusion

PC2Seg combines label-space consistency and feature-space contrastive learning for semi-supervised semantic segmentation, while using negative sampling to improve pixel contrastive learning efficiency and effectiveness. Experiments show improvements over existing methods across several semi-supervised segmentation benchmarks.

  • PC2Seg jointly uses feature-space contrastive learning and label-space consistency training for semi-supervised semantic segmentation.
  • Negative sampling techniques address efficiency and effectiveness challenges in pixel contrastive learning.
  • PC2Seg outperforms existing methods on several semi-supervised segmentation benchmarks.

A. Additional Implementation Details

The implementation follows DeepLab-related hyper-parameter defaults or public-code settings, with specified learning-rate, regularization, and crop-size choices across datasets.

  • DeepLab hyper-parameters use recommended defaults or values from public code.
  • The learning rate is linearly annealed from 0.007 to 0 during training.
  • Weight decay is 1e-4 for ResNets and 4e-5 for Xception.
  • Training crops use 513 × 513 inputs, while evaluation crops vary across VOC, COCO, and Cityscapes.Evaluation crop sizes are 513 × 513 for VOC, 641 × 641 for COCO, and 1,025 × 2,049 for Cityscapes.

B. Additional Ablation Results

Additional ablations examine projection-layer dimension and compare label-consistent-only training with the full joint regularization. The results support the benefit of combining label consistency with feature contrastive learning.

  • Projection Layer Dimension: A projection-layer dimension of 128 is used for the main VOC 1/8 ResNet-50 experiments.Alternative dimensions do not improve results with other hyper-parameters fixed and may require re-tuning.
  • Label Consistent Only: The joint label-consistent and feature-contrastive regularization performs better than label-consistent regularization alone.The label-consistent-only version removes the pixel contrastive loss while keeping the remaining setup unchanged.
  • Label Consistent Only: 71.79% validation mIoU is achieved by label-consistent-only PC2Seg versus 72.11% mIoU for full PC2Seg on Cityscapes 1/8.

C. Training Time and Computational Cost

Semi-supervised training takes longer than the supervised baseline, while negative sampling substantially reduces the computational cost of pixel contrastive learning. A t-SNE visualization also shows clearer semantic-class separation for semi-supervised methods than for the supervised baseline.

  • Training Time: PC2Seg training takes roughly 80 minutes versus 38 minutes for the supervised baseline in VOC ResNet-50 experiments.The label-consistent-only version takes around 75 to 80 minutes, comparable to PseudoSeg at about 80 minutes.
  • Feature Visualization: Semi-supervised methods show clearer semantic-class separation than the supervised baseline in t-SNE feature-space visualizations.The visualization uses 10,000 sampled data points with perplexity 40 and includes Decoder and Conv5 features.
  • Computational Cost: Sampling 200 negative pixels reduces the overall floating-point operations by about 5 times compared with using all pixels as negatives.The sampled computation uses 111M contrastive operations plus 379M sampling operations, versus 2,428M contrastive operations without sampling.

D. Visualization

PC2Seg is visualized through complementary feature-space and predicted-mask views. The feature visualizations indicate greater separability than the supervised baseline, with the joint variant slightly widening some category margins.

  • The appendix visualizes PC2Seg using t-SNE plots of feature spaces and predicted segmentation masks.These complementary views cover both learned representations and segmentation outputs.
  • Both label-consistent-only and joint contrastive-consistent variants produce more separable feature spaces than the supervised baseline.
  • The joint contrastive-consistent variant slightly increases margins between a few decoder-feature categories relative to the label-consistent-only variant.
  • The predicted masks contain both success and failure cases of PC2Seg, with detailed interpretations provided in the figure captions.
Loading 2108.09025v1…