PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals

Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Luc Van Gool

arXiv:2102.06191v3cs.CVcs.LG

TL;DR

Unsupervised semantic segmentation seeks dense semantic image representations without costly ground-truth labels, but prior work largely addressed limited datasets or relied on proxy tasks and end-to-end clustering. The paper introduces a two-step framework that mines object mask proposals as a mid-level prior and uses contrastive optimization to learn pixel embeddings. These embeddings support fully unsupervised segmentation on PASCAL and promising transfer to other datasets, while the method remains limited by salient-object estimators that retrieve only a limited number of objects per image.

  • Problem

    Unsupervised semantic segmentation remains relatively unexplored on challenging datasets traditionally used for supervised learning, despite the importance and annotation cost of dense semantic labels.

  • Method

    A two-step framework mines object mask proposals with unsupervised saliency and uses them as a mid-level prior in a contrastive objective for learning pixel embeddings.

  • Results

    The learned embeddings can be directly clustered into semantic groups on PASCAL under the fully unsupervised setting and show promising transfer results on other datasets.

  • Takeaways & Limitations

    Using object-level mid-level information provides a useful basis for self-supervised representation learning and supports semantic segmentation beyond proxy-task or end-to-end clustering approaches.

  • Takeaways & Limitations

    The method's object mask proposals come from a salient-object estimator that can retrieve only a limited number of objects per image.

Abstract

from arXiv · show

Being able to learn dense semantic representations of images without supervision is an important problem in computer vision. However, despite its significance, this problem remains rather unexplored, with a few exceptions that considered unsupervised semantic segmentation on small-scale datasets with a narrow visual domain. In this paper, we make a first attempt to tackle the problem on datasets that have been traditionally utilized for the supervised case. To achieve this, we introduce a two-step framework that adopts a predetermined mid-level prior in a contrastive optimization objective to learn pixel embeddings. This marks a large deviation from existing works that relied on proxy tasks or end-to-end clustering. Additionally, we argue about the importance of having a prior that contains information about objects, or their parts, and discuss several possibilities to obtain such a prior in an unsupervised manner. Experimental evaluation shows that our method comes with key advantages over existing works. First, the learned pixel embeddings can be directly clustered in semantic groups using K-Means on PASCAL. Under the fully unsupervised setting, there is no precedent in solving the semantic segmentation task on such a challenging benchmark. Second, our representations can improve over strong baselines when transferred to new datasets, e.g. COCO and DAVIS. The code is available.

1. Introduction

The paper targets fully unsupervised semantic segmentation by learning pixel embeddings without ground-truth labels. Its two-step framework uses unsupervised object-mask proposals as a mid-level prior for contrastive representation learning, enabling clustering or fine-tuning.

  • Motivation: Semantic segmentation assigns dense semantic labels to images, but conventional state-of-the-art methods require labor-intensive pixel-wise annotations.The task supports applications including autonomous driving, augmented reality, and human-computer interaction.
  • Objective: The paper learns pixel-level representations without ground-truth so pixels can be directly clustered into semantic groups using K-Means.The representations can also be fine-tuned when limited annotated examples are available.
  • Method: The proposed framework first mines object mask proposals with unsupervised saliency, then uses them as a prior in a contrastive objective for learning pixel embeddings.Embeddings from pixels belonging to the same object are pulled together and contrasted against embeddings from other objects.
  • Contribution: The method emphasizes object-level mid-level information rather than proxy tasks, end-to-end clustering, or low-level visual cues.The authors identify mid-level visual priors as useful for self-supervised representation learning.
  • Results: The paper reports fully unsupervised semantic segmentation on challenging PASCAL and promising transfer results on other datasets.It presents this as a first attempt on traditionally supervised, challenging benchmarks.

2. Related Work

Prior unsupervised segmentation methods mainly used end-to-end clustering or boundary-derived representations and were evaluated in limited settings. This paper distinguishes itself by using object-mask proposals as a mid-level prior rather than a proxy task.

  • Unsupervised semantic segmentation: Earlier fully unsupervised segmentation methods used end-to-end mutual-information objectives but were applied mainly to small datasets with narrow visual domains.Examples included separating sky from vegetation in satellite imagery.
  • Unsupervised semantic segmentation: Boundary-derived segments were used by some methods to learn pixel embeddings, but their suitability for offline clustering into discrete semantic labels remained unclear.Those evaluations focused on semantic segment retrieval requiring annotated training data.
  • Unsupervised semantic segmentation: Some prior boundary-based approaches also relied on additional supervision, including ImageNet pretraining and boundary annotations.
  • Representation learning: Representation-learning research commonly uses pretext tasks such as colorization, context prediction, jigsaw solving, clustering, optical flow, and inpainting.
  • Representation learning: For semantic segmentation, related pixel-level methods likewise relied on proxy tasks, whereas this paper explicitly avoids using a proxy task.

3. Method

MaskContrast learns pixel embeddings through a two-step pipeline: object mask proposals provide a mid-level grouping prior, which is incorporated into contrastive learning to separate pixels from different objects. The resulting embeddings support clustering and transfer across datasets.

  • 3. Method: The learned pixel embeddings can be directly clustered into semantic groups or further fine-tuned when annotated examples are available.This decouples feature learning from clustering and supports both fully unsupervised and semi-supervised use.
  • 3. Method: MaskContrast first identifies object regions as a mid-level visual prior, then uses them to learn pixel embeddings with contrastive learning.The prior groups pixels likely to belong together before feature learning, reducing dependence on network initialization.
  • 3.1. A Mid-Level Visual Prior for Grouping Pixels: Shared pixel ownership defines the prior: pixels in the same mask are grouped together, avoiding direct reliance on proxy-task outputs.This bottom-up strategy identifies mid-level visual groups before producing a complete semantic segmentation.
  • 3.1. A Mid-Level Visual Prior for Grouping Pixels: Unsupervised saliency estimation generates object mask proposals from unlabeled images, while supervised saliency estimation is also explored as an alternative.The paper reports training unsupervised and supervised saliency estimators on MSRA and DUTS, respectively, then predicting masks on PASCAL.
  • 3.3. MaskContrast: Learning Pixel Embeddings by Contrasting Salient Objects: The contrastive objective pulls pixels toward the embedding of their object and pushes them away from embeddings of other objects.The embedding function maps pixels to a normalized D-dimensional hypersphere, and object representations are formed from mean pixel embeddings.
  • 3.3. MaskContrast: Learning Pixel Embeddings by Contrasting Salient Objects: Background pixels are excluded from contrastive learning, so a separate linear head predicts saliency masks to prevent image-wide embedding collapse.The pixel-wise loss is applied to foreground pixels, while mask prediction regularizes the feature space.

4. Experiments

Experiments evaluate MaskContrast on PASCAL and test its ablations, comparisons, direct clustering, retrieval, and transfer to COCO and DAVIS.

  • Experimental setup: The evaluation uses PASCAL, COCO, and DAVIS-2016 to test segmentation performance and transferability of pixel embeddings.PASCAL provides the main benchmark; COCO and DAVIS-2016 assess transfer to novel scenes and video object segmentation.
  • Ablation studies: 58.4% versus 6.5% shows that MaskContrast produces more useful linear-separable semantic embeddings than saliency features alone on PASCAL.The saliency estimator separates salient objects from background, whereas MaskContrast structures embeddings so visually similar objects are closer and dissimilar objects farther apart.
  • Ablation studies: 45.0% to 58.4% for MoCo and 53.1% to 61.0% for supervised pre-training show gains beyond the initialization models.Additional supervision further improves results, reaching 62.2% or 63.9% with a supervised saliency estimator depending on initialization.
  • Ablation studies: Salient object masks outperform hierarchical segmentation regions because the latter are often too small to represent an object or part.The ablation supports using object-level information as the mid-level prior.
  • Ablation studies: Augmented positives, a memory bank, and momentum-updated negative encoders each improve performance, while the reported results show limited sensitivity to temperature and negative count.Augmentations add invariances, the memory bank better estimates negatives, and momentum updates enforce memory-bank consistency.
  • Comparison with prior methods: MaskContrast outperforms proxy-task, clustering, contrastive-learning, and boundary-based methods in the reported PASCAL comparisons.The comparisons attribute the advantage to shared pixel ownership, pixel-level contrastive learning, and higher-level saliency information rather than proxy objectives or low-level cues.
  • Clustering and retrieval: MaskContrast pixel embeddings can be directly clustered with K-Means into semantic groups on PASCAL and support strong segment retrieval.Retrieval evaluates seven PASCAL classes quantitatively and also reports results across all 21 classes.
  • Transfer learning: 55.4% versus 45.0% on PASCAL and 45.0% versus 35.2% on COCO show transfer gains over MoCo v2.The representations also transfer to DAVIS-2016, where they handle viewpoint changes, occlusions, and other natural image augmentations.

5. Discussion and Limitations

MaskContrast learns semantically meaningful pixel embeddings by combining object-level priors with contrastive learning, while its current saliency-based proposals limit the number of objects it can retrieve per image.

  • Discussion: MaskContrast uses a two-step framework with a mid-level visual prior to prevent representations from latching onto low-level image features.The framework separates prior construction from feature learning and contrasts with proxy-task and end-to-end clustering approaches.
  • Limitations: The method’s main limitation is that its salient-object estimator retrieves only a limited number of objects per image.The paper proposes exploring alternative proposal sources, including additional sensory data, for images containing many objects.
  • Method: The method groups salient-object pixels into prototypes, contrasts same-object pixels with other-object pixels, and adds an auxiliary BCE loss to prevent collapse.The pseudocode also includes query/key networks, a prototype queue, momentum updates, and temperature-scaled contrastive logits.
  • Method: The contrastive objective combines positive logits, negative logits, and cross-entropy over the designated positive index.Positive logits compare query embeddings with object prototypes, while negative logits compare them with queued prototypes.

B. Pseudo-code

The pseudocode specifies that MaskContrast uses publicly available saliency estimators with their default hyperparameters to obtain training masks.

  • Pseudo-code: Saliency masks are generated with public code from existing saliency estimators using their default hyperparameter settings.The authors state that a PyTorch implementation and pre-computed saliency masks will be publicly available.
  • Pseudo-code: The implementation and pre-computed saliency masks are intended for public release.This supports reproducibility of the proposal-generation stage.

C. Pre-training

The pre-training comparison uses established pretrained models and task-specific feature extraction strategies, with MaskContrast implemented using a dilated ResNet-50 and DeepLab-v3 head.

  • Pre-training: Most compared models use pretrained weights released by their original authors.The setup covers several prior representation-learning methods and uses their available checkpoints where possible.
  • Baselines: Co-Occurrence, Colorization, CMP, and IIC follow their original or adapted strategies for linear classification and K-Means evaluation.Colorization features are selected from intermediate layers, using the best reported layer for evaluation.
  • Baselines: Contrastive-learning baselines use ImageNet-pretrained ResNet-50 weights from instance discrimination, SWAV, MoCo v2, and InfoMin.When multiple released variants existed, the authors selected the best available model.
  • Pre-training: Contrastive-learning models were pretrained only on ImageNet because further pretraining on PASCAL produced no substantial improvement.Dilated convolutions are applied in the final residual block to obtain dense predictions.
  • MaskContrast: MaskContrast uses a dilated ResNet-50 with a DeepLab-v3 head and separate heads for pixel embeddings and saliency-mask prediction.During linear evaluation, the final layer is replaced with a randomly initialized 1 x 1 convolutional layer.

D. Linear Classifier

Linear evaluation trains a single 1 x 1 convolutional layer with stochastic gradient descent under a fixed 60-epoch schedule.

  • Linear Classifier: The linear classifier is trained for 60 epochs using batches of 16 and the complete training set.The optimization uses stochastic gradient descent with momentum 0.9, weight decay 0.0001, and initial learning rate 0.1.
  • Linear Classifier: The learning rate is reduced to 0.01 after 40 epochs.Increasing training time or modifying the learning rate did not improve the results.

E. Clustering

The evaluation obtains semantic class assignments by clustering pixel or object representations, then aligns clusters with ground-truth classes to compute mean IoU. Overclustering is additionally used to assess semantic structure when direct cluster matching is difficult on PASCAL.

  • K-Means assignments are aligned with ground-truth classes using the Hungarian algorithm before mean IoU is computed over five runs.This evaluation procedure follows the referenced strategy and averages results across five runs.
  • Contrastive-based methods upsample cluster assignments to image resolution before Hungarian matching.
  • MaskContrast clusters mean embeddings of foreground objects using L2-normalized features and assigns each object’s pixels the resulting label.A saliency estimation head identifies the background class before Hungarian matching.
  • Overclustering permits many predicted clusters to map to one target cluster, helping characterize semantic structure when target clusters are unknown and imbalanced PASCAL is difficult.The passage states that Table S1 reports these results and that accuracy improves, but provides no complete value here.

F. Semi-Supervised Learning

The semi-supervised experiments average results across three random data splits and compare an ImageNet-pretrained baseline with MaskContrast fine-tuning across label regimes.

  • Semi-supervised results are averaged over three randomly sampled splits for each setup.
  • The ImageNet-pretrained baseline uses a dilated-convolution ResNet-50 backbone with batch size 8, momentum 0.9, and learning rate 0.004.The learning rate was selected through grid search.
  • MaskContrast fine-tuning uses learning rate 0.004 with 5%, 12.5%, and 100% labels, but 0.001 for most layers with 1% and 2% labels.At 1% and 2%, the final convolutional layer uses learning rate 0.1 and batch-normalization statistics are frozen.

G. Qualitative Results

The qualitative comparison evaluates linear classifiers trained on learned representations against several self-supervised baselines. The visual results support semantically meaningful pixel embeddings.

  • Figure S1 compares linear-classifier outputs from MaskContrast representations with instance discrimination, SWAV, and MoCo v2.
  • MaskContrast representations are shown with either an unsupervised or supervised saliency estimator.
  • The qualitative comparison supports the claim that the learned pixel embeddings capture semantically meaningful information.
Loading 2102.06191v3…