PaperlayerView exact revision on arXiv

Source-linked AI summary

Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrast

Ye Du, Zehua Fu, Qingjie Liu, Yunhong Wang

arXiv:2110.07110v3cs.CV

TL;DR

Image-level WSSS struggles because CAMs derived from image tags leave a gap between classification and the pixel-level supervision needed for complete masks. The paper introduces cross-view and intra-view pixel-to-prototype contrast with mining strategies, improving WSSS baselines while preserving inference cost.

  • Problem

    Image-level WSSS lacks accurate pixel locations, and CAMs therefore provide incomplete supervision for segmentation.

  • Method

    Weakly-supervised pixel-to-prototype contrast uses cross-view and intra-view regularization, with prototype mining and hard-pixel sampling to shape dense representations.

  • Results

    The method consistently improves two strong WSSS baselines by large margins and achieves top performance on the standard benchmark.

  • Takeaways & Limitations

    The approach narrows the classification–segmentation supervision gap and can be incorporated into existing WSSS models without extra inference burden.

  • Takeaways & Limitations

    Prototype estimation by clustering may not fully use image tags and generally requires over-clustering because resulting clusters may not match true categories.

Abstract

from arXiv · show

Though image-level weakly supervised semantic segmentation (WSSS) has achieved great progress with Class Activation Maps (CAMs) as the cornerstone, the large supervision gap between classification and segmentation still hampers the model to generate more complete and precise pseudo masks for segmentation. In this study, we propose weakly-supervised pixel-to-prototype contrast that can provide pixel-level supervisory signals to narrow the gap. Guided by two intuitive priors, our method is executed across different views and within per single view of an image, aiming to impose cross-view feature semantic consistency regularization and facilitate intra(inter)-class compactness(dispersion) of the feature space. Our method can be seamlessly incorporated into existing WSSS models without any changes to the base networks and does not incur any extra inference burden. Extensive experiments manifest that our method consistently improves two strong baselines by large margins, demonstrating the effectiveness. Specifically, built on top of SEAM, we improve the initial seed mIoU on PASCAL VOC 2012 from 55.4% to 61.5%. Moreover, armed with our method, we increase the segmentation mIoU of EPS from 70.8% to 73.6%, achieving new state-of-the-art.

1. Introduction

Image-level WSSS must infer pixel locations from image tags, while CAM-based seeds often cover only discriminative object parts. The proposed pixel-to-prototype contrast supplies pixel-level supervision through cross-view and intra-view learning, improving CAMs, seeds, and segmentation without inference changes.

  • Motivation: Image tags provide object-category presence but not accurate locations, creating a supervision gap between classification and pixel-level segmentation.CAMs estimate target regions but typically produce incomplete and inaccurate seeds for pseudo-ground-truth generation.
  • Method: The method performs pixel-to-prototype contrast across views and within each view, enforcing semantic consistency and feature-space compactness and dispersion.Cross-view learning uses transformed views, while intra-view learning pulls pixels toward positive prototypes and away from negative prototypes.
  • Method: Semi-hard prototype mining and hard pixel sampling reduce inaccurate contrasts caused by over-activated and under-activated CAM regions.These strategies address unreliable pseudo-mask regions, particularly for intra-view contrast.
  • Results: The approach can be integrated into existing WSSS models with additional training-time projectors, without modifying base networks or adding inference burden.The supplied passages report consistent gains for SEAM and EPS in initial seed quality and segmentation performance.
  • Contribution: The work introduces weakly-supervised pixel-to-prototype contrast to narrow the classification–segmentation gap by giving pixels supervision from reliable class prototypes.This is presented as the paper’s central contribution under weak supervision.
  • Results: The method improves CAM quality and subsequent segmentation masks, surpasses baseline models by large margins, and achieves top performance on the standard benchmark.The contribution summary reports consistent improvements from applying contrastive learning within and across image views.

2. Related Work

Related work addresses incomplete CAM-derived pseudo masks through additional features, saliency, cross-image semantics, and consistency constraints. The paper connects prototype-based pixel contrast with dense contrastive learning and cross-view consistency regularization for weakly supervised segmentation.

  • Image-level WSSS: Image-level WSSS commonly generates pseudo masks from CAMs, but CAMs highlight only the most discriminative object regions and therefore produce incomplete masks.The central challenge is the supervision gap between classification and segmentation.
  • Image-level WSSS: Prior approaches narrow the supervision gap using multi-level or accumulated features, cross-image semantics, sub-categories, saliency maps, and CAM consistency constraints.These methods add forms of supervision or regularization while retaining the pseudo-mask training pipeline.
  • Contrastive Learning: Supervised and dense contrastive learning motivate feature-space objectives that encourage intra-class compactness and inter-class dispersion for dense prediction.The related work also discusses clustering-based contrastive learning and prototype-oriented objectives.
  • Consistency Regularization: Consistency regularization enforces semantic or distributional agreement across perturbations, including image augmentation and network perturbation.The paper positions its cross-view pixel-to-prototype contrast as feature semantic consistency regularization across views.
  • Consistency Regularization: Unlike common semi-supervised segmentation settings, cross-view consistency regularization has been less studied for weakly supervised segmentation.The paper applies this consistency idea across different views of each image.

3. Methodology

The method adds pixel-to-prototype contrast to WSSS by shaping projected pixel embeddings with CAM-derived category prototypes across views and within each view. Cross-view regularization enforces semantic consistency, while intra-view contrast promotes compactness and uses mining strategies to mitigate inaccurate pseudo-label contrasts.

  • 3.1. Preliminary: CAMs generate pixel-wise pseudo masks by assigning each pixel the category with the highest activation.The method follows CAM generation with pixel-wise argmax before segmentation training.
  • 3.2. Pixel-to-Prototype Contrast: Pixel-to-prototype contrast regularizes each pixel toward its positive category prototype and away from negative prototypes in a projected feature space.Prototypes are representative category embeddings estimated from high-confidence CAM pixels.
  • 3.3. Prototype Estimation: Category prototypes are weighted averages of the top-K CAM-confidence projected embeddings, with smaller K indicating higher confidence.Prototype computation can select high-confidence pixels across the training batch to capture global dataset context.
  • 3.4. Cross-view Contrast: Cross-view contrast borrows prototypes and pseudo labels between spatially transformed views to impose semantic consistency symmetrically.The source and target views are encoded separately, while the source feature map and CAM receive the same spatial transformation.
  • 3.5. Intra-view Contrast: Intra-view contrast uses same-view prototypes to encourage intra-class compactness and inter-class dispersion of pixel representations.Applying it naively can degrade performance because weakly supervised pseudo labels may be inaccurate.
  • 3.5. Intra-view Contrast: Semi-hard prototype and pixel mining mitigate incorrect contrasts by selecting difficult negatives and pixels that are dissimilar to their corresponding prototypes.For prototypes, the method selects from the hardest negative prototypes; for pixels, hardness is based on dot-product similarity to the assigned prototype.

4. Experiment

Experiments on PASCAL VOC 2012 show that pixel-to-prototype contrast improves CAM seeds, pseudo masks, and downstream segmentation across SEAM and EPS baselines. Ablations support the contributions of cross-view and intra-view components, while prototype estimation remains robust to K.

  • Experimental Setup: The method is evaluated on PASCAL VOC 2012 using SEAM and EPS as strong baselines, with mIoU reported for seeds, pseudo masks, and segmentation.The experiments use the augmented training set and report validation and test performance for semantic segmentation.
  • Seed and Pseudo Mask Evaluation: 6.0% and 7.2% mIoU gains over SEAM are reported for the initial seed and seed+CRF, respectively, on the VOC training set.The method also produces CAMs with more complete object coverage and more accurate boundaries than SEAM.
  • Seed and Pseudo Mask Evaluation: 5.9% mIoU separates the method from the best-performing AdvCAM counterpart on the initial seed under IRN refinement settings.The reported gain is attributed primarily to the method’s high-quality initial seed rather than the refinement procedure.
  • Segmentation Performance: 3.2% and 1.7% mIoU improvements over SEAM are obtained on the PASCAL VOC validation and test sets using DeepLab-LargeFOV.With DeepLab-LargeFOV and ResNet101, EPS equipped with the method achieves 72.3% and 73.5% mIoU on the validation and test sets.
  • Segmentation Performance: The method achieves new state-of-the-art performance with DeepLab-ASPP on PASCAL VOC 2012 and works on both simple and challenging scenes.Qualitative results are presented on the VOC 2012 validation set.
  • Ablation Study: With all components, SEAM mIoU increases by 6.13% on the train set and 5.87% on the validation set.Cross-view consistency terms first improve performance, while sample mining mitigates the slight drop introduced by intra-view pixel-to-prototype contrast.
  • Ablation Study: K=32 gives the best prototype-estimation performance, and the method is robust across a wide range of K values.Smaller K uses more confident pixels, whereas larger K may include pixels from other categories.
  • Ablation Study: Adding different spatial transformations is not substantially effective, yielding an observation similar to SEAM.The evaluated transformations include rotation, translation, and horizontal flipping.

5. Conclusion

The paper concludes that weakly supervised pixel-to-prototype contrast supplies pixel-level supervision for image-level WSSS through cross-view and intra-view contrast. Experiments validate its superiority, while future work targets prototype estimation and broader segmentation settings.

  • Conclusion: Weakly supervised pixel-to-prototype contrast narrows the supervision gap and improves image-level WSSS by providing pixel-level supervisory signals.The method performs contrast across views and within a single view to regularize feature consistency and class structure.
  • Conclusion: The method’s superiority is validated through extensive experiments, with future work including new prototype-estimation methods and fully supervised or semi-supervised segmentation.The conclusion identifies broader application scenarios as a future direction.
Loading 2110.07110v3…