PaperlayerView exact revision on arXiv

Source-linked AI summary

Causal Unsupervised Semantic Segmentation

Junho Kim, Byung-Kwan Lee, Yong Man Ro

arXiv:2310.07379v1cs.CVcs.AIcs.LG

TL;DR

Unsupervised semantic segmentation leaves open what to cluster and how to reach the intended granularity. CAUSE uses frontdoor adjustment to connect pre-trained features to semantic groups through a discretized concept clusterbook and concept-wise self-supervised learning. It reports state-of-the-art performance across datasets without pixel annotations.

  • Problem

    Unsupervised semantic segmentation lacks a clear target for what and how to cluster, including the granularity needed to group pixel representations into semantic concepts.

    Source: §1 · pp. 1–2

  • Method

    CAUSE applies frontdoor adjustment by constructing a discretized concept clusterbook as a mediator, then using its prototypes for concept-wise self-supervised semantic grouping.

    Source: §1 · pp. 2–3

  • Results

    CAUSE achieves state-of-the-art performance in unsupervised semantic segmentation across various datasets without pixel annotations.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    The findings bridge causal perspectives into unsupervised prediction and improve segmentation quality without pixel annotations.

    Source: §5 · p. 10

    The method assigns each patch feature only its closest concept prototype and simplifies the likelihood relation under a uniform-distribution assumption.

    Source: §3.3 · pp. 5–6

Abstract

from arXiv · show

Unsupervised semantic segmentation aims to achieve high-quality semantic grouping without human-labeled annotations. With the advent of self-supervised pre-training, various frameworks utilize the pre-trained features to train prediction heads for unsupervised dense prediction. However, a significant challenge in this unsupervised setup is determining the appropriate level of clustering required for segmenting concepts. To address it, we propose a novel framework, CAusal Unsupervised Semantic sEgmentation (CAUSE), which leverages insights from causal inference. Specifically, we bridge intervention-oriented approach (i.e., frontdoor adjustment) to define suitable two-step tasks for unsupervised prediction. The first step involves constructing a concept clusterbook as a mediator, which represents possible concept prototypes at different levels of granularity in a discretized form. Then, the mediator establishes an explicit link to the subsequent concept-wise self-supervised learning for pixel-level grouping. Through extensive experiments and analyses on various datasets, we corroborate the effectiveness of CAUSE and achieve state-of-the-art performance in unsupervised semantic segmentation.

1 INTRODUCTION

Unsupervised semantic segmentation avoids costly pixel-level labels, but it leaves unresolved what to cluster and at what granularity. CAUSE frames this ambiguity through causal inference, using a discretized concept clusterbook to connect pre-trained features with concept-wise self-supervised pixel grouping.

1 INTRODUCTION

  • Prior USS approaches use visual consistency, multi-view equivalence, saliency priors, or self-supervised pre-trained features to support unsupervised segmentation.Recent methods use contrastive learning to maximize feature correspondence.Source: §1 · p. 1
  • USS lacks a well-defined target for what and how to cluster, making it difficult to achieve the desired semantic granularity from pixel representations.The introduction notes that body parts such as head, torso, hands, and legs should be grouped as person, a target previous methods struggled to achieve.Source: §1 · pp. 1–2
  • CAUSE constructs a discretized concept clusterbook M of prototypes spanning possible concepts and granularity levels, then uses its indices to select positive and negative features for concept-wise self-supervised learning.The segmentation head consolidates prototypes into broader target categories.Source: §1 · p. 2
  • The paper casts the clusterbook as a mediator in a frontdoor-adjustment framing, linking pre-trained features T to semantic groups Y while addressing backdoor paths induced by clustering indeterminacy U.The introduction describes the resulting two-step tasks as discrete representation learning followed by do-calculus with self-supervised learning.Source: §1 · pp. 2–3
  • CAUSE reports state-of-the-art unsupervised semantic segmentation results across various datasets, using a frontdoor-adjustment framework with a concept clusterbook and concept-wise self-supervised learning.The framework is presented as working without pixel annotations.Source: §1 · p. 3

2 RELATED WORK

Prior USS work has used several sources of inductive bias and, more recently, self-supervised representations. CAUSE aligns with representation-based methods while emphasizing that the clustering target itself remains underdefined.

2 RELATED WORK

3 CAUSAL UNSUPERVISED SEMANTIC SEGMENTATION

CAUSE frames unsupervised semantic segmentation as a frontdoor-adjustment problem with an unobserved clustering indetermination. It uses a discretized concept clusterbook as mediator, then trains concept-matched pixel features with concept-wise self-supervision.

3.1 DATA GENERATING PROCESS FOR UNSUPERVISED SEMANTIC SEGMENTATION

  • Unobserved clustering indetermination is treated as a confounder, so CAUSE uses a mediator to link pre-trained features to semantic groups without observing that confounder.The two tasks are constructing the clusterbook and learning semantic groups conditioned on features and the mediator.Source: §3.1 · p. 4

3.2 CONSTRUCTING CONCEPT CLUSTERBOOK FOR MEDIATOR

  • The concept clusterbook discretizes pre-trained features into prototypes at multiple granularities, using modularity to account for patch locality and connectivity.Modularity jointly considers prototype assignments and feature affinities; trace operations and a temperature-scaled hyperbolic tangent address small values during optimization.Source: §3.2 · p. 5

3.3 ENHANCING LIKELIHOOD FOR SEMANTIC GROUPS

  • A frozen pre-trained model supplies features that are matched to their closest clusterbook prototypes and passed through a transformer segmentation head to produce concept-matched features.The head uses cross-attention between quantized prototype embeddings and input features, followed by an MLP projection.Source: §3.3 · p. 5
  • Concept-wise self-supervision selects positive and negative examples by prototype similarity, using softer positive relaxation to consolidate subdivided concepts and tighter negative relaxation.The stated thresholds are 0.3 for positive selection and 0.1 for negative selection.Source: §3.3 · p. 6
  • A pixel-level concept bank accumulates out-of-batch features for additional comparison pairs, extending self-supervised learning beyond local views.For each prototype, the bank randomly removes half its features and samples half new features at each iteration.Source: §3.3 · p. 6
  • The training objective increases the likelihood of semantic groups through concept-wise self-supervised learning, with an exponential-moving-average teacher-student structure also used.The framework uses log-probability to avoid near-zero values from repeated multiplication.Source: §3.3 · pp. 6–7

4 EXPERIMENTS

CAUSE is evaluated on COCO-Stuff, Cityscapes, and Pascal VOC against unsupervised segmentation baselines. It generally outperforms recent methods, extends to object-centric segmentation and larger category sets, and shows gains across most categories.

4.1 EXPERIMENTAL DETAILS

  • Evaluation uses frozen DINO features and benchmarks COCO-Stuff, Cityscapes, and Pascal VOC, with inference steps matched to STEGO and HP.The datasets use 27 mid-level categories for COCO-Stuff and Cityscapes and 21 classes for Pascal VOC.Source: §4.1 · pp. 7–8

4.2 VALIDATING CAUSE

  • CAUSE generally outperforms recent unsupervised segmentation frameworks and achieves state-of-the-art results without external information.The comparison reports mIoU and pAcc on multiple datasets.Source: §4.2 · p. 8
  • CAUSE also achieves superior linear-probing results to baselines, indicating competitive dense representation quality learned without supervision.Table 2 reports the linear-probing comparison.Source: §4.2 · p. 8, Table 2 · p. 8
  • CAUSE can distinguish foreground objects from background for Pascal VOC without external information by adjusting its positive relaxation.The cited results are for Pascal VOC 2012.Source: §4.2 · p. 8
  • CAUSE retains superior performance with larger category sets and alternative self-supervised encoders, with a particular pAcc advantage on COCO-171.The comparisons include DINOv2, iBOT, MSN, and MAE, as well as COCO-81 and COCO-171.Source: Table 3 · p. 8, §4.2 · p. 9
  • CAUSE segments concepts across most COCO-Stuff categories more effectively than STEGO and HP, which struggle with Thing categories in complex scenes.The categorical analysis uses IoU results on a logarithmic scale and reports improved visual quality for targeted semantic groupings.Source: §4.2 · p. 9, Figure 5 · p. 9
  • Ablations identify relaxation settings as important, show concept counts saturating after 2048, and examine the concept bank, CRF, and modularity maximization.The studies cover six factors, including positive and negative relaxation and the clusterbook discretization method.Source: §4.2 · p. 9, Table 4 · p. 9

5 CONCLUSION

CAUSE addresses indeterminate clustering targets in unsupervised semantic segmentation with frontdoor adjustment, a concept-clusterbook mediator, and concept-wise self-supervision. Experiments report state-of-the-art performance without pixel annotations.

5 CONCLUSION

  • CAUSE uses frontdoor adjustment to connect a concept-clusterbook mediator with concept-wise self-supervision, reporting state-of-the-art unsupervised segmentation without pixel annotations.The framework targets indeterminate clustering targets in unsupervised semantic segmentation.Source: §5 · p. 10

A EXPANSION OF RELATED WORKS

Prior work uses unsupervised objectives, cross-image correspondences, saliency, and pretrained self-supervised representations for segmentation. CAUSE frames indeterminate clustering targets causally and details implementations for its clusterbook and transformer head.

A EXPANSION OF RELATED WORKS

  • Earlier unsupervised segmentation methods use mutual-information objectives, cross-image correspondences, saliency, and pretrained self-supervised representations as pseudo-labels or priors.The paper positions CAUSE alongside feature-enhancement methods while emphasizing indeterminate clustering targets and a causal framing that bridges discretized representation learning and contrastive learning.Source: A EXPANSION OF RELATED WORKS · pp. 15–16

B DETAILED IMPLEMENTAION OF CAUSE

  • The implementation appendix provides details for CAUSE and points to trained CAUSE-MLP and CAUSE-TR parameters and visualizations in supplementary code documentation.It presents these resources as supporting identifiable and reproducible performance.Source: B DETAILED IMPLEMENTAION OF CAUSE · p. 16

B.1 MAXIMIZING MODULARITY

  • For large prototype counts, modularity computation uses trace-property rearrangement to reduce computational burden, while a temperature-scaled hyperbolic tangent rescales small values.The appendix notes that directly multiplying small assignment-matrix values can make modularity too small for effective optimization.Source: §B.1 · pp. 16–17

B.2 TRANSFORMER-BASED SEGMENTATION HEAD

  • CAUSE-TR uses a single-layer transformer decoder with self-attention, cross-attention, and an FFN; its output dimension is reduced to 90 to avoid degradation associated with dimensions above 100.The appendix describes vector quantization against concept features and learnable positional embeddings.Source: §B.2 · p. 17

B.3 ANCHOR SELECTION

The implementation reduces anchor-point computation through stochastic sampling, selects concept positives and negatives by cluster similarity, and maintains a bounded concept bank. Dataset-specific resolutions and augmentation are also specified.

B.3 ANCHOR SELECTION

  • Randomly sampling one feature point per 4 × 4 window selects 6.25% of patch features for anchor points.The passage presents this as a way to reduce computation on resource-constrained hardware.Source: §B.3 · p. 17

B.4 POSITIVE & NEGATIVE CONCEPT SELECTION

  • Positive and negative concept features are selected by comparing their distances to the closest concept prototype, with hard negative relaxation set to 0.1.The closest prototype is identified by maximizing cosine similarity; the passage also describes filtering with positive and negative distance thresholds.Source: §B.4 · p. 17

B.5 CONCEPT BANK

  • The concept bank re-samples 50% of segmentation features and stores at most 100 features per prototype, totaling 100 × 2048 features.Before re-sampling, 50% of the bank is randomly discarded; the bank is intended to provide diverse candidates for positive and negative sampling.Source: §B.5 · p. 17

B.6 IMAGE RESOLUTION AND AUGMENTATION

  • For COCO-Stuff and Cityscapes, training uses five crops at 0.5 crop ratio resized to 224 × 224, while inference resizes images to 320 × 320 along the minor axis before center cropping.CAUSE-TR uses 320 × 320 for training; the described augmentation uses horizontal flips only, unlike the additional transforms employed by the referenced methods.Source: §B.6 · pp. 17–18

C ADDITIONAL EXPERIMENTS

The appendix adds qualitative segmentation examples and baseline comparisons because the main manuscript could not include a comprehensive set of visual results.

C ADDITIONAL EXPERIMENTS

  • Page limits prevented the main manuscript from including a comprehensive set of visual results across multiple datasets.The additional section supplies examples primarily from four datasets and compares them with baseline methods.Source: C ADDITIONAL EXPERIMENTS · p. 18
  • The additional experiments provide qualitative unsupervised segmentation examples primarily from four datasets, alongside baseline comparisons.Source: C ADDITIONAL EXPERIMENTS · p. 18
  • The added visual results address the manuscript’s stated lack of comprehensive multi-dataset examples.Source: C ADDITIONAL EXPERIMENTS · p. 18

C.1 ADDITIONAL QUALITATIVE RESULTS

The additional qualitative results compare CAUSE with unsupervised segmentation baselines on scene and object-centric datasets, with adjusted positive relaxation for datasets containing a background class.

C.1 ADDITIONAL QUALITATIVE RESULTS

  • Additional visual results compare CAUSE with recent DINO-feature baselines on COCO-Stuff and Cityscapes.The experimental setup remains consistent with the main manuscript.Source: §C.1 · p. 18
  • PASCAL VOC, COCO-81, and COCO-171 visualizations cover object-centric segmentation and include an additional background class.The positive relaxation values are 0.2, 0.4, and 0.55, respectively, while negative relaxation remains 0.1.Source: §C.1 · p. 18
  • Positive relaxation is adjusted across the three object-centric datasets to account for the coarsely merged background class.The stated purpose is to help distinguish the integration of background concepts.Source: §C.1 · p. 18

C.2 FAILURE CASES

CAUSE can produce inadequate segmentations, including noisy outcomes in complex scenes and incorrect assignments for some object instances. The discussed remedies involve trade-offs and future work on features and clustering.

C.2 FAILURE CASES

  • The authors describe unsupervised segmentation as challenging because it combines absent supervision with dense pixel-level prediction, and note that CAUSE has failure cases.Source: §C.2 · p. 18
  • Increasing the clusterbook concept count k may help with complex-scene noise, but trades off performance on easier examples.The authors identify preprocessing and multi-scale feature extraction as future directions for improving detailed dense prediction.Source: §C.2 · p. 18
  • Some object instances receive incorrect cluster assignments, which the authors attribute primarily to image-level self-supervised pretraining limitations or incomplete clustering.They propose exploring foundation networks designed for dense prediction and unsupervised clustering algorithms.Source: §C.2 · p. 18

C.3 CONCEPT ANALYSIS IN CLUSTERBOOK

The clusterbook is examined by retrieving image regions that share an anchor concept index, with the concept distance matrix used to merge regions. The analysis uses an anchor selected from 2,048 concepts and shows retrieval examples for Person and Animal on COCO-Stuff.

C.3 CONCEPT ANALYSIS IN CLUSTERBOOK

  • The analysis selects an anchor index from the clusterbook’s 2,048 concepts and retrieves image regions assigned to that same index.This tests what concepts the discretized mediator represents in the representation space.Source: §C.3 · p. 19
  • The concept distance matrix is used to merge regions whose concept distance from the anchor exceeds a positive relaxation of 0.3.The merging accounts for wider image regions using pixel-level clusterbook indices.Source: §C.3 · p. 19
  • Retrieval examples use Person and Animal categories and CAUSE predictions on COCO-Stuff.The examples are presented in Figure 11.Source: §C.3 · p. 19

D DISCUSSIONS AND LIMITATIONS

Pre-trained representations can struggle to segment intricate details in complex scenes, while CAUSE combines discrete concept representations with concept-wise self-supervised learning for pixel-level grouping. The discussion identifies heuristic relaxation selection as a limitation and calls for more dynamic criteria for specialized, complex contexts.

Bootstrapping Pre-trained Models

  • CAUSE uses high-level pre-trained features for pixel-level semantic grouping by combining discrete concept representations with concept-wise self-supervised learning.The authors frame this as bootstrapping pre-trained knowledge through a causal approach.Source: D DISCUSSIONS AND LIMITATIONS · p. 19
  • STEGO, HP, and ReCo+ struggle to segment intricate details, especially with densely packed objects, complex backgrounds, or small objects.The discussion attributes this to a mismatch between pre-training tasks and the granularity required for fine-grained segmentation; qualitative results span Coco-Stuff, COCO-171, Cityscapes, PASCAL VOC, and COCO-81.Source: D DISCUSSIONS AND LIMITATIONS · pp. 19, 21, Figure 6 · p. 20, Figure 7 · p. 20

Heuristically Static Hyper-Parameter

  • Selecting positive and negative relaxation criteria for the clusterbook is heuristic, and grid-searching values from 0.1 to 0.7 requires effort.The discussion identifies dataset-specific tuning as challenging.Source: D DISCUSSIONS AND LIMITATIONS · p. 19
  • The authors identify dynamic criterion selection for small objects and intricate scenes as a future direction.These are examples of specialized and complex image contexts for which static criteria may be inadequate.Source: D DISCUSSIONS AND LIMITATIONS · p. 19

Related papers

Loading 2310.07379v1…