PaperlayerView exact revision on arXiv

Source-linked AI summary

Causal Intervention for Weakly-Supervised Semantic Segmentation

Dong Zhang, Hanwang Zhang, Jinhui Tang, Xiansheng Hua, Qianru Sun

arXiv:2009.12547v2cs.CV

TL;DR

WSSS pseudo-masks derived from image-level labels have ambiguous boundaries because context acts as a confounder. The paper models these causalities and introduces CONTA, whose context adjustment improves pseudo-masks and achieves state-of-the-art results.

  • Problem

    Existing WSSS methods struggle to generate accurate pixel-level pseudo-masks from image-level labels, while the causal reasons for ambiguous boundaries remain insufficiently explained.

  • Method

    CONTA uses a Structural Causal Model and an approximate backdoor adjustment to perform virtual context intervention and generate higher-quality pseudo-masks.

  • Results

    CONTA improves pseudo-masks by 2.0% mIoU on average and achieves 66.1% mIoU on the PASCAL VOC 2012 validation set and 66.7% on its test set.

  • Takeaways & Limitations

    CONTA improves prevailing WSSS methods by removing confounding context bias from image-level classification and producing better pseudo-masks for subsequent segmentation.

  • Takeaways & Limitations

    The framework avoids the extremely challenging task of building a generative model for context-to-image relationships, constraining the intervention approach to this approximation.

Abstract

from arXiv · show

We present a causal inference framework to improve Weakly-Supervised Semantic Segmentation (WSSS). Specifically, we aim to generate better pixel-level pseudo-masks by using only image-level labels -- the most crucial step in WSSS. We attribute the cause of the ambiguous boundaries of pseudo-masks to the confounding context, e.g., the correct image-level classification of "horse" and "person" may be not only due to the recognition of each instance, but also their co-occurrence context, making the model inspection (e.g., CAM) hard to distinguish between the boundaries. Inspired by this, we propose a structural causal model to analyze the causalities among images, contexts, and class labels. Based on it, we develop a new method: Context Adjustment (CONTA), to remove the confounding bias in image-level classification and thus provide better pseudo-masks as ground-truth for the subsequent segmentation model. On PASCAL VOC 2012 and MS-COCO, we show that CONTA boosts various popular WSSS methods to new state-of-the-arts.

1 Introduction

WSSS generates pixel-level pseudo-masks from image-level labels, but context-driven correlations create ambiguous boundaries and incomplete object regions. CONTA addresses this confounding context through causal modeling and virtual intervention, improving pseudo-masks and segmentation performance.

  • Pipeline: WSSS trains a multi-label classifier, extracts class-specific seed areas with CAM, and expands them into pseudo-masks for supervised segmentation training.The pseudo-masks are the key intermediate target in the prevailing pipeline.
  • Pseudo-mask problems: Object co-occurrence causes ambiguous boundaries, such as confusion between people and horses when horses commonly appear with riders.
  • Pseudo-mask problems: Foreground-background co-occurrence produces incomplete backgrounds, exemplified by floor regions being misclassified as sofa.
  • Pseudo-mask problems: Context-dependent foreground parts produce incomplete foregrounds because classifiers favor discriminative regions such as a car’s wheel over its window.
  • Causal motivation: Context acts as a confounder that creates spurious pixel-label associations and motivates estimating P(Y ∣do(X)) instead of P(Y ∣X).The do-operation seeks causality between pixels and labels without the confounding effect.
  • Contribution: CONTA uses a Structural Causal Model and backdoor adjustment to perform virtual intervention, improving pseudo-masks by 2.0% mIoU on average and reaching 66.1%/66.7% mIoU on PASCAL VOC validation/test sets and 33.4% on MS-COCO validation.The method incorporates possible contexts into multi-label classification to generate better CAM seed areas.

2 Related Work

Related work frames WSSS around reducing annotation costs through pseudo-mask generation and segmentation training, while highlighting visual context and causal inference as relevant directions. CONTA explicitly combines context use with causal intervention.

  • Weakly-Supervised Semantic Segmentation: WSSS reduces fully supervised segmentation’s labeling cost by using image-level labels, with pseudo-masks expected to cover entire objects and preserve accurate boundaries.
  • Visual Context: Most WSSS models implicitly use visual context through enlarged receptive fields, while prior work also explicitly aligns same-class foreground features and contrasts other features.
  • Causal Inference: Causal inference seeks causal effects, including removal of spurious bias and disentanglement of desired model effects; this paper adopts Pearl’s Structural Causal Model for WSSS.

3 Context Adjustment

CONTA models context as a confounder in WSSS classification and applies causal intervention to improve CAM seeds and pseudo-masks. It approximates unobserved contexts with iterative, class-specific mask updates before training a segmentation model.

  • 3.1 Structural Causal Model: The SCM represents causalities among pixel-level images X, context prior C, image-level labels Y, and the X-specific mediation M.Its links include C → X, C → M ← X, and X → Y ← M.
  • 3.1 Structural Causal Model: The context confounds X and Y through the backdoor path X ← C → M → Y, correlating non-causal pixels with labels and degrading pseudo-masks.The model attributes problematic CAM seeds to this overlooked context pathway.
  • 3.2 Causal Intervention via Backdoor Adjustment: CONTA estimates P(Y ∣do(X)) with backdoor adjustment, cutting C → X and averaging predictions across stratified contexts rather than performing physical intervention.This gives X an opportunity to incorporate every context according to a prior P(c).
  • 3.2 Causal Intervention via Backdoor Adjustment: Because C is unobservable, CONTA approximates its set with class-specific average masks and iteratively establishes the missing contexts.Each context entry is an average mask for images of a corresponding class, while M is an X-specific mask represented as a linear combination of these entries.
  • 3.2 Causal Intervention via Backdoor Adjustment: CONTA iterates classification, CAM-based pseudo-mask generation, and segmentation training, treating the procedure as an EM algorithm over observed X and missing C.At t = 0, M_t is empty; later segmentation masks provide additional context estimates for subsequent iterations.
  • 3.2 Causal Intervention via Backdoor Adjustment: The method approximates the context expectation at feature level, moving the outer context sum into a single feed-forward computation using normalized weighted geometric means.The approximation combines f(X,c) using P(c), with uniform P(c)=1/n and softmax-normalized similarities α_i.

4 Experiments

Experiments evaluate CONTA across datasets, baselines, ablations, and qualitative comparisons. Results show improved pseudo-masks and segmentation performance, with gains across multiple WSSS models and state-of-the-art comparisons.

  • Experimental Setup: CONTA was evaluated quantitatively and qualitatively on PASCAL VOC 2012 and MS-COCO using CAM, pseudo-mask, and segmentation-mask mIoU.Training used only image-level class labels, while segmentation masks were evaluated on validation and test sets.
  • Experimental Setup: CONTA was deployed on SEAM, IRNet, DSRG, and SEC to assess applicability across popular WSSS baselines.The experiments used the official settings of the respective methods and DeepLab-v2 as the common segmentation model.
  • Ablation Study: Round 3 achieved the best mIoU on CAM, pseudo-mask, and segmentation mask, so the following CONTA experiments used three rounds.Qualitative pseudo-masks also showed progressively clearer boundaries than the baseline.
  • Ablation Study: 1.1%, 2.3%, and 1.8% mIoU gains were obtained on CAM, pseudo-mask, and segmentation mask, respectively, when concatenating Mt at block-5 versus the baseline.Block-5 performed best among the tested feature-map locations, while block-4 and block-5 were slightly higher than earlier blocks.
  • Ablation Study: Using both pseudo-masks and segmentation masks to establish the confounder set improved performance, with segmentation masks producing the larger gain.The authors suggest that segmentation masks provide a higher-quality approximation because the trained segmentation model smooths pseudo-masks.
  • Effectiveness on Different Baselines: 0.9%, 2.0%, and 2.0% were the average mIoU improvements on CAM, pseudo-mask, and segmentation mask across different PASCAL VOC 2012 baselines.CONTA with SEAM reached 56.2% CAM and 66.1% segmentation-mask mIoU, while CONTA with IRNet reached 67.9% pseudo-mask mIoU.
  • Comparison with State-of-the-arts: 66.1% and 66.7% mIoU were achieved on the PASCAL VOC 2012 validation and test sets by CONTA with SEAM and ResNet-38, surpassing prior best results by 1.2% and 1.0%.On MS-COCO, CONTA with SEC and VGG-16 achieved 23.7% validation mIoU, surpassing the previous best by 1.3%.
  • Qualitative Results: CONTA improved object location and boundary predictions in qualitative comparisons, but bicycle and plant remained failure cases.The authors attribute these failures possibly to obtaining segmentation masks from 8× down-sampled feature maps.

5 Conclusion

The paper frames WSSS pseudo-mask errors as consequences of context confounding and uses causal intervention to address them. CONTA improves existing WSSS methods, while its confounder approximation remains a stated limitation.

  • Conclusion: CONTA uses causal intervention based on backdoor adjustment to remove context confounding from image-level classification.The method approximates the unobserved confounder with class-specific information and generates improved pseudo-masks.
  • Conclusion: CONTA promotes prevailing WSSS methods to new state-of-the-art performance.The conclusion presents this as the overall empirical outcome of the proposed causal framework.
  • Conclusion: The approximation of the context confounder is ill-posed, motivating future work on confounder discovery and incorporation of observable expert knowledge.These are identified as directions for improving CONTA's confounder modeling.

Broader Impact

The work reports potential benefits for fairness and context-robust object segmentation, while acknowledging misuse risks that require responsible use.

  • The method may improve fairness in weakly-supervised semantic segmentation by reducing potential discrimination from models that cater to majority patterns.The passage gives gender, racial, and religious discrimination as examples of potential harms from unfair AI.
  • The method may enable accurate object segmentation without photographing objects across every possible context.The passage illustrates this with segmenting a car on the road without training images of cars under every context.
  • Misuse could include segmenting minority groups for malicious purposes, so the technique should be used for appropriate purposes.

Appendix for “Causal Intervention for Weakly-Supervised Semantic Segmentation”

The appendix provides derivations, implementation details, supplementary ablations, and additional segmentation-mask visualizations supporting the paper’s main method and analyses.

  • The appendix derives backdoor adjustment for the proposed structural causal model.This corresponds to Section 1.
  • It derives the normalized weighted geometric mean used in the method.This corresponds to Section 2.
  • It documents detailed implementations for different baseline models and reports supplementary ablation studies.These materials appear in Sections 3 and 4.
  • It includes additional visualizations of segmentation masks.These visualizations appear in Section 5.

1 Derivation of Backdoor Adjustment for the Proposed Causal Graph

The appendix derives the interventional distribution P(Y ∣do(X)) for the proposed causal graph by applying three do-calculus rules and modeling image-specific context as a function of images and classes.

  • The derivation uses do-calculus Rules 1–3 to manipulate observations and interventions in the proposed causal graph.The rules cover insertion/deletion of observations, action/observation exchange, and insertion/deletion of actions.
  • The derivation begins by expanding P(Y ∣do(X)) over the confounder c using the law of total probability.Equation A4 introduces the summation over c.
  • Rule 3 yields Eq. A5 by removing the intervention on c, while Rule 2 yields Eq. A6 by replacing intervention with observation under the stated graph conditions.
  • The image-specific context representation M is defined as M = f(X, c), enabling the transition from Eq. A7 to Eq. A8.

2 Normalized Weighted Geometric Mean

The appendix derives the normalized weighted geometric mean approximation used to move the outer confounder-weighted sum from the prediction level into the feature level.

  • NWGM approximates ∑c P(Y ∣X, M)P(c) with P(Y ∣X, M = ∑c f(X, c)P(c)).This moves the outer sum over c into a weighted feature representation.
  • For the positive class term, the derivation starts from a confounder-weighted ratio of exponentiated scores.The expression uses the positive score s1(c), the competing score s2(c), and P(c).
  • The derivation converts the weighted score expression into exponentiated weighted sums and applies the softmax definition and prior results.Equations A13–A15 follow the derivation in [68].
  • Because the positive score is implemented as a linear model, Eq. (3) in the main paper can compute M_t+1.

3 More Implementation Details

The appendix specifies model-specific backbones, preprocessing, optimization, and mask-refinement settings used to deploy CONTA across WSSS baselines.

  • Backbone: CONTA was deployed with different backbone architectures and output strides across the four WSSS models.The implementations used ResNet-38 with stride 8, ResNet-50 or ResNet-101 with stride 16, and VGG-16 with stride 8.
  • Setting: Input preprocessing varied by implementation, using fixed crops of 448 × 448, 512 × 512, or 321 × 321 with zero padding when needed.One setting additionally randomly rescaled the longest edge between 448 and 768 before cropping.
  • Training Details: Training configurations differed substantially in optimizer, iteration or epoch count, batch size, learning rate schedule, and regularization.The settings included Adam for 8 epochs, SGD for 8,000 or 10,000 iterations, polynomial decay, momentum, dropout, and weight decay.
  • Hyper-parameters: CAM and mask processing used fully connected CRF refinement, while seed expansion employed AffinityNet with γ = 5, β = 8, and t = 256 in one configuration.CAM thresholds were generally set to 16, with values of 4 and 24 used to amplify or weaken background activation.

4 More Ablation Study Results

Additional ablations show that CONTA’s iterative procedure performs best after two or three rounds, depending on the baseline and dataset.

  • PASCAL VOC 2012: IRNet+CONTA and SEC+CONTA achieved their best performance at round = 3, whereas DSRG+CONTA achieved its best mIoU at round = 2 on PASCAL VOC 2012.IRNet+CONTA reached 48.8% on CAM, 67.9% on pseudo-mask, and 65.3% on segmentation mask as its second-best mIoU results.
  • PASCAL VOC 2012: 48.8% on CAM, 67.9% on pseudo-mask, and 65.3% on segmentation mask were reported for IRNet+CONTA on PASCAL VOC 2012.These were described as IRNet+CONTA’s second-best mIoU results in the appendix.
  • MS-COCO: SEAM+CONTA, IRNet+CONTA, and SEC+CONTA achieved top mIoU at round = 3, while DSRG+CONTA performed best at round = 2 on MS-COCO.The appendix reports the same round-dependent pattern across the four baseline models.
  • MS-COCO: 28.7% on CAM, 35.2% on pseudo-mask, and 33.4% on segmentation mask were the reported IRNet+CONTA mIoU scores on MS-COCO.These scores were described as the best among the IRNet+CONTA results on MS-COCO.

5 More Visualizations

Visualizations indicate that SEAM+CONTA produces more accurate and complete masks, particularly around thin object edges and parts.

  • Qualitative Results: SEAM+CONTA masks were reported as more accurate and more complete for examples including cows, horses, birds, people, and cars.The cited examples include a person lying next to a dog and a person standing next to cows.
  • Qualitative Results: SEAM+CONTA improved predictions for thin object edges or parts such as bird tails or heads, car regions, and people in cars.These examples highlight boundary improvements in visually narrow structures.
  • Figure A1: Figure A1 presents additional PASCAL VOC 2012 samples, with red rectangles marking regions improved by SEAM [63]+CONTA.The figure is intended to visualize further segmentation results.
Loading 2009.12547v2…