PaperlayerView exact revision on arXiv

Source-linked AI summary

Class Re-Activation Maps for Weakly-Supervised Semantic Segmentation

Zhaozheng Chen, Tan Wang, Xiongwei Wu, Xian-Sheng Hua, Hanwang Zhang, Qianru Sun

arXiv:2203.00962v1cs.CV

TL;DR

CAM-based WSSS can produce ambiguous pseudo masks because BCE permits nonexclusive class activation and shared partial responses. ReCAM reactivates BCE-trained CAM features with SCE using class-specific soft masks, and evaluations on PASCAL VOC and MS COCO report higher-quality masks with plug-and-play use across CAM variants. The method has reported scope boundaries involving transformer-based classification models and possible overfitting of its second FC layer to noisy backbone features.

  • Problem

    BCE-trained CAMs can contain false-positive pixels from confusing classes and false-negative object regions, limiting pseudo-mask quality in WSSS.

  • Method

    ReCAM applies each class’s soft CAM to backbone features, then learns an additional FC layer with SCE and extracts reactivated CAMs.

  • Results

    ReCAM generates better masks than vanilla CAM across single-label and multi-label images, remains effective with IRN, and achieves 54.8% on VOC versus 55.6% for AdvCAM while running 160× faster.

  • Takeaways & Limitations

    ReCAM is a simple plug-in seed-generation module that can improve CAM masks and combine with direct, iterative, or IRN-based mask-generation options.

  • Takeaways & Limitations

    The implementation does not support transformer-based classification models, and ReCAM’s second FC layer may overfit noisy features from a poor backbone.

Abstract

from arXiv · show

Extracting class activation maps (CAM) is arguably the most standard step of generating pseudo masks for weakly-supervised semantic segmentation (WSSS). Yet, we find that the crux of the unsatisfactory pseudo masks is the binary cross-entropy loss (BCE) widely used in CAM. Specifically, due to the sum-over-class pooling nature of BCE, each pixel in CAM may be responsive to multiple classes co-occurring in the same receptive field. As a result, given a class, its hot CAM pixels may wrongly invade the area belonging to other classes, or the non-hot ones may be actually a part of the class. To this end, we introduce an embarrassingly simple yet surprisingly effective method: Reactivating the converged CAM with BCE by using softmax cross-entropy loss (SCE), dubbed \textbf{ReCAM}. Given an image, we use CAM to extract the feature pixels of each single class, and use them with the class label to learn another fully-connected layer (after the backbone) with SCE. Once converged, we extract ReCAM in the same way as in CAM. Thanks to the contrastive nature of SCE, the pixel response is disentangled into different classes and hence less mask ambiguity is expected. The evaluation on both PASCAL VOC and MS~COCO shows that ReCAM not only generates high-quality masks, but also supports plug-and-play in any CAM variant with little overhead.

1. Introduction

WSSS commonly derives pseudo masks from CAMs trained with image-level labels, but BCE can produce class-overlapping activations and incomplete object coverage. ReCAM addresses these flaws by reactivating BCE-trained CAM features with SCE to learn more class-specific responses.

  • Motivation: WSSS uses image-level class labels to reduce annotation cost, then trains a classifier, extracts CAM masks, and learns segmentation from pseudo labels.Image-level labels are cheaper than scribbles or bounding boxes, but the classification model is the root of CAM quality.
  • Problem: BCE with sigmoid activation permits nonexclusive class activation, causing false-positive pixels from confusing classes and false-negative pixels from shared partial activation.Because BCE does not penalize activation on other classes, multiple classes can respond within the same receptive field.
  • Motivation: SCE enforces class exclusion by jointly improving the ground-truth logit and penalizing competing logits.Its softmax denominator creates contrastive learning across classes.
  • Evidence: SCE models produce higher-CAM-mIoU masks while maintaining classifier quality for 80-class models, with especially significant gains for five hoofed-animal classes.The reported superiority is nearly maintained on validation images.
  • ReCAM: ReCAM extracts each class CAM as a soft mask, applies it to backbone features to create single-label class-specific features, and uses them for reactivation.This branches a multi-label feature map into class-specific feature sets before learning with SCE.

2. Related Works

Prior CAM improvements modify feature discovery, classification supervision, or mask refinement. ReCAM instead adds SCE-based reactivation while preserving the activation function and requiring only one forward pass at inference.

  • Seed Generation: Vanilla CAM generates seeds by scaling backbone feature maps with class-specific FC weights, then averaging channels, normalizing spatially, and thresholding.GAIN, erasing methods, and Score-CAM modify or replace parts of this seed-generation process.
  • Mask Generation: Mask-refinement methods propagate seed regions to semantically similar neighboring pixels using affinity-based random walks.PSA predicts adjacent-pixel affinities, while related methods vary the transition-matrix design.
  • Other CAM Improvements: Other CAM methods learn intra-class boundaries, pseudo fine-grained labels, transformation consistency, or information-bottleneck-based classifiers.These approaches target feature boundaries, finer supervision, augmentation consistency, or classification-model design.
  • ReCAM: ReCAM adds an SCE-based loss without removing the activation function and feeds each test image forward once.On PASCAL VOC 2012, its total overhead over vanilla CAM is 0.6 hours versus 8 hours of inference cost for RIB.

3. Preliminaries

CAM trains a multi-label classifier with BCE, extracts class-specific activation maps from backbone features and FC weights, and converts them into pseudo masks for subsequent segmentation training. ReCAM can be inserted into the seed-generation stage while retaining multiple mask-generation choices.

  • CAM: Vanilla CAM trains a multi-label classifier with global average pooling and an FC prediction layer using BCE over image-level class labels.The labels indicate whether each foreground class is present in the image.
  • CAM: After convergence, CAM for class k is extracted from the pre-GAP feature maps using the corresponding FC classification weights.The resulting class map is denoted CAM_k(x), or M_k.
  • Pseudo Masks: Pseudo masks may be formed by thresholding CAM, refining it with IRN, iteratively refining it through the classifier, or cascading iterative refinement with IRN.These options are also illustrated with ReCAM plugged into the pipeline.
  • Semantic Segmentation: The segmentation model is then trained in a fully supervised manner using the generated all-class masks as pseudo labels.The segmentation objective includes K foreground classes plus the background class.
  • Implementation: The implementation uses DeepLab variants with ResNet-101 and also evaluates UperNet with a Swin Transformer backbone.These are implementation choices for the segmentation models.

4. Class Re-Activation Maps (ReCAM)

ReCAM reactivates a converged BCE-based CAM model with SCE on class-specific features, producing class-exclusive representations and ReCAM masks.

  • Scope: The method was not implemented with transformer-based classification models, which the authors identify as future work.This limits the reported implementation scope to the model families evaluated in the paper.
  • ReCAM Pipeline: ReCAM first extracts normalized CAM soft masks from a conventional BCE classifier and applies each mask to backbone features.This produces class-specific feature maps while retaining the original multi-label classification stage.
  • ReCAM Pipeline: Each class-specific feature map receives a single class label and trains a second fully connected classifier with SCE loss.The SCE gradients update both the second classifier and the backbone.
  • ReCAM Pipeline: The reactivation objective retains BCE optimization while adding SCE, and updated soft masks are used during learning.The BCE loss updates FC1 so it can continue producing the masks required for feature extraction.
  • ReCAM Pipeline: After reactivation, ReCAM is extracted for each class and can optionally be refined with AdvCAM or IRN before training segmentation models.The resulting refined maps serve as pixel-level labels for semantic segmentation.
  • Justification: BCE vs CE: SCE provides class-exclusive learning because its gradients jointly encourage the positive class and suppress the confusing negative class.For a much-lower positive logit, SCE gradients approach magnitude 1, whereas BCE gradients remain below 0.5; when logits are similarly confusing, SCE acts on both classes.

5. Experiments

Experiments on PASCAL VOC and MS COCO show that ReCAM improves pseudo-mask quality over CAM, remains effective with refinement and different CAM variants, and adds limited computational overhead.

  • Experimental Setup: Experiments use PASCAL VOC 2012 and MS COCO 2014, with VOC expanded to 10,582 training images and MS COCO containing 80 object classes.The implementation uses a ResNet-50 backbone and trains the additional FC layer for four epochs.
  • Qualitative Analysis: ReCAM improves both false-negative and false-positive mask errors, while a failure case remains under occlusion or similar object-background colors.The qualitative analysis covers masks before segmentation-model training and identifies failures such as confusion between dogs and human hands.
  • Pseudo-Mask Quality: ReCAM outperforms CAM by 6% in pseudo-mask mIoU on VOC, with nearly maintained gains after training semantic segmentation models.The advantage is reported consistently on both VOC and MS COCO and remains when IRN refinement is added.
  • Efficiency and Generality: IRN increases computational cost by about 4.5× and AdvCAM by 160× over vanilla ReCAM on ResNet-50, while the best WSSS performance consistently uses IRN.ReCAM is also reported to support plug-and-play use with different CAM variants, including saliency-based methods.

6. Conclusions

The paper identifies BCE as the source of conventional CAM flaws and proposes ReCAM, which reactivates BCE-based models with SCE. Experiments on VOC and MS COCO show consistent advantages and generality.

  • ReCAM addresses false-positive and false-negative flaws in conventional CAM by reactivating a BCE-based model with SCE.

Supplementary materials

The supplementary materials provide additional toy experiments, quantitative WSSS results, analyses of false-positive and false-negative pixels, sensitivity studies, and derivations.

  • The supplementary materials include toy experiments and additional quantitative WSSS results.
  • They analyze false-positive and false-negative pixels, pseudo-mask quality across λ values, learning-rate sensitivity, and SCE/BCE derivations.

A. More Details about Toy Experiments

The toy experiment studies five hoofed-animal classes in MS COCO using carefully selected single-class-labeled images to compare BCE and SCE classifiers.

  • The experiment uses horse, sheep, cow, elephant, and bear images from MS COCO, ignoring other co-occurring classes.
  • It contains 6,340 training images and 3,001 validation images for these selected cases.
  • The selected images are labeled with a one-hot class label despite possible co-occurring objects.

B. More WSSS Results (DeepLabV3+)

Supplementary results evaluate WSSS with DeepLabV3+ and DeepLabV2, comparing CAM or ReCAM seeds, refinement choices, and FC weights across VOC and MS COCO.

  • Table S1 reports mIoU% for DeepLabV3+ on VOC and MS COCO using CAM or ReCAM seeds followed by mask refinement.
  • Table S2 compares DeepLabV2 mIoU% when ReCAM uses different FC weights.
  • The DeepLabV2 results distinguish direct ReCAM pseudo masks from masks further refined with IRN.

D. Statistics of Two Flaws

Table S3 analyzes true positives, false positives, and false negatives in seed masks, showing that ReCAM substantially reduces false negatives relative to CAM, with threshold-dependent trade-offs.

  • D. Statistics of Two Flaws: Table S3 distinguishes true positives, object and background false positives, and false negatives in the seed masks.The reported percentages of TP, FN, and FP (bg) sum to 100% in each column.
  • D. Statistics of Two Flaws: Compared with CAM at threshold 0.21, ReCAM significantly decreases false-negative pixels but slightly increases object and background false positives.
  • D. Statistics of Two Flaws: Increasing ReCAM’s threshold to 0.26 reduces both false-negative and false-positive pixels, but lowers mIoU from 54.8% to 53.8%.
  • D. Statistics of Two Flaws: On MS COCO, ReCAM reduces both false positives and false negatives while achieving its best performance on the more challenging dataset.

E. λ on MS COCO

The supplementary experiments examine λ and learning-rate sensitivity, while documenting the two-stage ReCAM training and its loss-gradient analysis.

  • E. λ on MS COCO: The hyperparameter λ balances the BCE and SCE terms, with 0.1 optimal on MS COCO and large values such as 2 causing substantial performance drops.
  • E. λ on MS COCO: Large learning rates destabilize CAM training and can produce NaN loss, whereas ReCAM is comparatively insensitive to the learning-rate value.
  • E. λ on MS COCO: ReCAM reuses weights from a pretrained BCE model rather than training its two fully connected layers from scratch.
  • E. λ on MS COCO: The supplementary material derives gradients for BCE and SCE and provides Algorithm 1 for the ReCAM training pipeline.

J. More Qualitative Results

Additional visualizations compare CAM and ReCAM soft and binary masks, refined masks, and final DeepLabV2 segmentation results on VOC and MS COCO.

  • J. More Qualitative Results: Figure S3 provides qualitative comparisons of CAM and ReCAM heatmaps and 0-1 masks on the VOC training set.
  • J. More Qualitative Results: Figure S4 visualizes masks from different CAM variants after refinement with IRN on VOC and MS COCO.
  • J. More Qualitative Results: Figure S5 visualizes DeepLabV2 semantic segmentation masks produced from seeds generated by different methods and subsequently refined by IRN.
  • J. More Qualitative Results: The qualitative examples include categories such as motorbike, boat, person, sofa, horse, dog, bear, potted plant, bottle, table, giraffe, sandwich, and bus.
Loading 2203.00962v1…